Prerequisites

Before going into the Product of Experts (PoE), we need to clarify some important mathematical foundations and concepts first. In this article, it is assumed that you are familiar with basic probabilities, calculus and linear algebra. Prior knowledge of matrices, vectors, partial derivatives, probabilities and statistics is required.

What is a Product of Experts?

A Product of Experts (PoE) models data by multiplying together several simpler probability distributions and re-normalizing, so that a data point is considered acceptable only when every expert finds it plausible: a single expert assigning it near-zero probability is enough to rule it out. The easiest way to understand this is to compare it with its opposite: the mixture model. A mixture model combines several probability distributions by taking a weighted sum of them. Given n individual models p_m(d), the mixture assigns to a data vector d the probability

$$\begin{equation} p_{\text{mix}}(\mathbf{d}) = \sum_{m=1}^{n} \pi_m , p_m(\mathbf{d}), \qquad \sum_{m=1}^{n}\pi_m = 1, \quad \pi_m \geq 0, \end{equation}$$

where the weights πm say how much each component contributes. Because a mixture is an average, a data point is considered likely if some component finds it likely — averaging tends to make the resulting distribution broader than its parts. This is precisely the model a Product of Experts is defined against: where a mixture sums (and blurs), a PoE multiplies (and sharpens). Keeping the mixture in mind makes the central idea of the PoE much easier to see. A simple way to put it is: let’s say we are searching for our dog that ran away and there are 10 possible places where he is.

Now, try to think of 2 experts (Expert A and Expert B) as two individuals that were on the street and saw the dog at a certain time.

After a brief talk, they give you some probabilities to help you find the dog:

In a mixture model, we combine the experts by taking the mean of their probabilities. For example in position 3:

$$\begin{equation} p_{\text{mix}}(3) = \frac{0.6 + 0.1}{2} = 0.35. \end{equation}$$

In a Product of Experts, we instead multiply the probabilities together and then normalize (divide by their sum, so the result adds up to 1). For example in position 3:

$$\begin{equation} 0.6 \times 0.1 = 0.06. \end{equation}$$

After doing the same for other positions, we do their sum (let's call it S) and divide each one for it.

$$\begin{equation} p_{\text{PoE}}(3) = \frac{0.06}{\text{S}} \approx 0.286. \end{equation}$$

By doing this for all the positions we have:

With this example, it is more clear to see the difference: while in the mixture there is a draw between Position 3 and 7, with room for suspecting the Position 5 as well (very tight probabilities for all of the positions), in the Product of Experts, we have a clear winner (Position 5). But why? The mixture ends up building a high probability when there is one expert confirming it, even though the other tells the exact opposite. The Product of Experts is built on the accordance between experts. Low probabilities given by the experts result in a high penalization. In this case, the two experts told ’Maybe the dog is in Place 5’ but they both told that the most probable place was another place (but they gave different places for that).

Why does PoE matter?

The dog example showed that multiplying experts produces a sharper answer than averaging them. Beyond being a neat trick, this behavior is exactly what makes a PoE useful in practice.

The first reason is that a PoE is a principled way to combine independent sources of knowledge. Rather than training a single model to capture everything about a problem, we can build or reuse several specialists, each responsible for one aspect, and multiply them together. Experts can be designed, trained, and improved separately, and new ones can be added or removed without rebuilding the whole system. It is kind of modular.

A second reason concerns uncertainty. Because experts are multiplied, a PoE becomes confident only where every expert agrees, and remains uncertain wherever they disagree. This is precisely the behavior we want from a well-calibrated system: confidence is earned by consensus across the evidence, not asserted by a single component. When decisions depend on trustworthy confidence estimates (for example in healthcare applications), the way a PoE concentrates certainty only where sources agree is a real advantage.

This power does come at a price. Multiplying and re-normalizing experts makes the distribution expensive to normalize and to sample from– the partition function problem we look at next– so a PoE trades easy training for sharpness, calibrated uncertainty, and modularity. The rest of this article is about how we pay that price in an efficient manner.

Mathematical Foundations

Now that we know what a PoE is, we need a few more tools to understand how it is trained. Among them are maximum likelihood and gradient ascent, the partition function, Bayes’ theorem, and Markov chains with Gibbs sampling.

Maximum Likelihood and Gradient Ascent

Given a model with parameters θ and a set of observed data vectors, maximum likelihood estimation (MLE) chooses the parameters that make the observed data as probable as possible under the model. Because probabilities of independent observations multiply, we work with the log-likelihood, which turns that product into a sum and is easier to handle:

$$\begin{equation} \mathcal{L}(\theta) = \sum_{\mathbf{d}\in\text{data}} \log p(\mathbf{d}\mid\theta). \end{equation}$$

We then look for the θ that maximizes L(θ).

Gradient ascent is the procedure that finds it. Starting from some initial parameters, we repeatedly take a small step in the direction of the gradient, since that is the direction of steepest increase:

$$\begin{equation} \theta \leftarrow \theta + \eta \frac{\partial \mathcal{L}(\theta)}{\partial \theta}, \end{equation}$$

where η > 0 is the learning rate.

This is the engine used to train a PoE: the whole learning algorithm amounts to computing the gradient of the log-likelihood and stepping uphill. As we will see, the difficulty is not the ascent itself but computing one part of that gradient, and that difficulty comes entirely from the partition function.

The Partition Function Z

When we combine models by multiplying their probabilities, the raw product does not automatically sum to 1, so it is not yet a valid probability distribution. The partition function Z is the normalizing constant that fixes this — it is the product summed (or, for continuous data, integrated) over every possible data vector:

$$\begin{equation} p(\mathbf{d}) = \frac{1}{Z}\prod_{m=1}^{n} p_m(\mathbf{d}\mid\theta_m), \qquad Z = \sum_{\mathbf{c}} \prod_{m=1}^{n} p_m(\mathbf{c}\mid\theta_m). \end{equation}$$

Dividing by Z guarantees that p(d) is a proper distribution.

The trouble is that Z is a sum over the entire data space, which is astronomically large in any realistic problem, so computing it directly is intractable. This single fact is the source of essentially every complication in training a PoE: because Z depends on the parameters, its derivative does not vanish, and it turns the learning gradient into something we cannot evaluate in closed form. The sampling machinery introduced later (Markov chains and Gibbs sampling) exists precisely to sidestep this problem.

Bayes' Theorem

Bayes’ theorem tells us how to update a belief about a hidden quantity H once we have observed data D. It relates the posterior P(H | D) to the prior P(H) and the likelihood P(D | H):

$$\begin{equation} P(H\mid D) = \frac{P(D\mid H),P(H)}{P(D)} \propto P(D\mid H),P(H) \end{equation}$$

where the denominator

$$P(D) = \sum_{H} P(D\mid H)$$

P(H) is a normalizing constant that makes the posterior probabilities sum to 1.

In words: posterior ∝ likelihood × prior.

In a PoE, Bayes' theorem is what lets each expert reason about its own hidden variables. In the training algorithm, for example, it is used to compute the posterior probability of which internal component of an expert is responsible for a given data point, a quantity we need in order to infer hidden states from the data and to generate reconstructions.

Markov Chains and Gibbs Sampling

A Markov chain is a sequence of random states in which the next state depends only on the current one, not on the full history. Under mild conditions a Markov chain converges to a fixed stationary (or equilibrium) distribution: after an initial settling-in period, the chain “forgets” where it started and visits states in stable long-run proportions, no matter the starting point. The clever part is that we can turn this around: if we design the chain so that this stationary distribution is the one we care about, then running the chain long enough produces samples from that distribution, turning the chain into a sampling machine for a distribution we could not sample directly.

Gibbs sampling is a Markov chain built for exactly this purpose. It draws samples from a complicated joint distribution by repeatedly resampling one variable at a time from its conditional distribution given the current values of all the others.

A simple way to picture this is a couple, Tom and Ana, deciding what to wear to a party, where they would like to match– both formal or both casual. We do not have a direct formula for what they will jointly wear, but we know how each one reacts to the other: if Tom is formal, Ana tends to go formal and if Tom is casual, Ana tends to go casual, and the same the other way around. Gibbs sampling says: do not try to solve the tangled joint problem at once. Instead, fix one person’s outfit and resample the other’s from their rule, then swap, using the updated value. Start anywhere: say Ana casual, Tom formal. Then resample Ana given Tom, then Tom given Ana, then Ana again, and so on. After enough back-and-forth rounds, the outfits we observe occur in exactly the right proportions of the true joint distribution, even though we never wrote that joint distribution down. We reached it through many simple one-at-a-time updates.

In a PoE this is especially convenient. The variables come in two groups: the visible ones (the data) and the hidden ones (the experts’ internal states), and the chain alternates between them just like Ana and Tom taking turns — resample the hidden states given the data, then resample the data given the hidden states. Better still, given the data, the experts’ hidden states are conditionally independent, so an entire group of variables can be resampled in parallel rather than one at a time.

This is how we obtain the samples (the “fantasy data” generated by the model itself) that we could not get from the partition function directly. Since computing Z blocks the direct route, we build a Markov chain instead. Those samples are what make the intractable term in the learning gradient estimable, closing the loop opened in the previous subsections. In principle this requires running the chain for many steps; remarkably, as we will see, a single step is often enough.

How PoE works

Having the tools, we will put them together to see how a PoE is actually trained. The plan is straightforward: write down what we want to maximize, compute its gradient, and deal with the one troublesome term that appears.

The learning objective

Training means choosing the experts’ parameters so that the observed data is as probable as possible under the model: this is maximum likelihood, introduced earlier. Since the PoE probability is a normalized product, we take its logarithm, which turns the product into a sum:

$$\begin{equation} \log p(\mathbf{d}) = \sum_{m=1}^{n} \log p_m(\mathbf{d}\mid\theta_m) - \log Z. \end{equation}$$

This objective has two parts. The first sum is easy: it is just each expert’s own log-probability on the data, and every expert is tractable by assumption. The second part, -log Z, is the troublemaker, because Z is the intractable sum over the entire data space. The objective is clean to write, but that -log Z term is what makes it hard to optimize.

The Gradient: The Data and The Fantasy

To run gradient ascent we differentiate the log-likelihood with respect to a parameter θ_m. The first sum differentiates easily. The interesting thing happens to the -log(Z) term: differentiating it turns it into an expectation over the model's own distribution. Averaging over the data as well, the gradient becomes a difference of two terms of the same shape:

Here, the intuition matters. The data term is averaged over the real data d, and it pushes the experts to raise the probability of what was actually observed. The fantasy term is the same quantity, but averaged over c (data generated by the model itself) and it pushes the experts to lower the probability of what the model currently believes. (In practice we estimate the data term on a mini-batch, which is a stochastic estimate of this average.) So learning is a tug-of-war: raise the probability of real data, lower the probability of the data the model dreams up. When the model’s fantasies match the real data, that is, when the model’s distribution matches the data distribution, the two averages cancel and learning stops, exactly the point where the model has captured the data distribution.

A simple way to picture this is a student learning from a textbook. The student has a book full of real physics problems (the data), and trains with two opposing moves each day. First, he reads the real problems and thinks “Yes... real problems look like this,” nudging his internal sense of a physics problem towards the textbook. Then he closes the book, invents his own problems from his current understanding (his fantasies) and says “these are just my inventions, trust them less,” nudging his sense away from his own guesses. Pull towards the real, push away from the dreamed-up: that is the whole tug-of-war.

Early on the student did not have the understanding of the real physics problems, so the in vented problems look nothing like real ones, and the two moves pull in clearly different directions, the student changes a lot. But as he improves, his inventions start to look just like real problems, because they have learned the pattern. At that point the “raise the real ones” move and the “lower my invented ones” move are acting on the same kind of thing, so they cancel, and learning stops. Not because the student gave up, but because their imagination now matches reality, which is exactly what it means to have captured the data distribution.

Crucially, this fantasy term is where the intractable log(Z) went: differentiating log(Z) is what produced the expectation over the model. The difficulty did not disappear; it turned into “we need samples from the model.

Generating Fantasy Data with Gibbs sampling

The problem is now concrete. The fantasy term is an expectation over the model's distribution, and an expectation can be estimated by averaging over samples. So we do not need Z after all, we just need samples of c from the model. But we cannot sample the PoE directly either (that would again require Z), which is exactly where Gibbs sampling comes in.

We start from the data and alternate: resample the experts' hidden states given the visible data, then resample the visible data given those hidden states, the same hidden to visible turn-taking as Tom and Ana did. Because the experts' hidden states are conditionally independent given the data, we resample the whole group in parallel rather than one at a time. Running this produces a reconstruction that serves as our fantasy sample, and plugging it into the fantasy term lets us estimate the gradient without ever touching Z.

Contrastive Divergence: Why one step is enough?

There is still one catch. In principle, Gibbs sampling must run to equilibrium (many steps) before its samples are truly from the model, and that is slow. This is where Hinton's surprising practical result lands: instead of running the chain to equilibrium, take just one Gibbs step starting from the real data, and use that one-step reconstruction as the fantasy sample. This is Contrastive Divergence (CD-1), and the update is simply the difference between correlations measured on the real data and on the one-step reconstruction:

$$\begin{equation} \Delta\theta_m \propto \left\langle \cdot \right\rangle_{\text{data}} - \left\langle \cdot \right\rangle_{\text{one-step reconstruction}} \end{equation}$$

Why does a single step suffice? Two intuitions from Hinton's [2] help. First, even one step already moves the chain towards equilibrium, so the update gets the right sign even if its magnitude is a little off. Second, because we start the chain at the data rather than from random noise, we do not need to wander far to get a useful contrast. The result is a training rule that is fast, local, and good enough in practice, which is exactly the "one step is often enough" promise made at the end of [4].

Picture the student once more. To make up a truly “fair” fantasy problem, one drawn properly from his current beliefs, he would have to brainstorm for hours, letting his mind wander far from any particular starting point until his ideas settle. That is the equilibrium the chain is supposed to reach, and it is slow. Contrastive Divergence takes a shortcut: instead of starting from a blank mind, the student starts from a real textbook problem and changes it just a little: tweak a number here, swap a scenario there. That lightly-edited problem is not a perfectly fair sample, but it is already a good enough “fantasy” to compare against the original: if his small edits made it worse, that tells him which way to adjust. One quick edit, starting from real data, is enough to get a useful contrast, no hours of wandering required.

With that, we have the full training recipe: the data term raises the probability of what is real, a single Gibbs step produces fantasy data, and their difference updates the experts. We are now ready to see it run.

A Practical Example of PoE (via RBM): The Dog's Behavior

In Section 2, the experts simply handed us their probabilities. But where do those numbers come from? In practice, a PoE learns them from data. Let’s build a real PoE that learns the dog’s hiding habits across a whole neighborhood, and then watch it complete a partial sighting and even dream up sightings of its own.

The concrete model is a Restricted Boltzmann Machine (RBM): the PoE mentioned earlier, with one expert per hidden unit. We train it with Contrastive Divergence (CD-1) — the “one Gibbs step is enough” trick from the previous sections.

You can run the code of the example by accessing the repository: PoE Example.

The Neighborhood and the Data

This time the street is a 12 × 12 grid of spots (144 in total), and the dog has 4 favourite hiding regions: one in each corner of the map. Each “day” we record a noisy sighting: a blurry blob of lit spots around one of the regions, with a few pixels randomly flipped so the data is never perfectly clean. We generate 4000 such days. Crucially, we never tell the model which region is which, or even that there are four of them because we want it to discover that structure on its own.

Building the Model to Find the Dog

We start with two small helpers. The sigmoid turns any number into a probability between 0 and 1, and sample turns a probability into an actual 0/1 state (a coin flip weighted by that probability):

import numpy as np
rng = np.random.default_rng(0)

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-np.clip(x,-30, 30)))

def sample(p): # turn probabilities into 0/1 states
    return (rng.random(p.shape) < p).astype(float)

Next, the model. There are 144 visible units (one per spot) and 9 hidden units: our nine experts. The weight matrix W connects spots to experts, and each group has a bias. These are the parameters that training will adjust:

N_VIS, N_HID = 144, 9 # 12x12 spots, 9 hidden "experts"
W = 0.01 * rng.standard_normal((N_VIS, N_HID))
b_v = np.zeros(N_VIS) # visible biases (per spot)
b_h = np.zeros(N_HID) #hidden biases (per expert)
lr = 0.05 # learning rate

Training with Contrastive Divergence: Learning the Dog's Habit

Each training step follows the same recipe as before, now on mini-batches of sightings. In the positive phase we push the real sightings up to the experts (the “data term”). In the negative phase we run a single Gibbs step to produce fantasy data (the “fantasy term”). The update is their difference. One detail worth noting: we drive the chain with a sampled hidden state h0, but use probabilities in the weight-update correlations, a small trick from Hinton’s work [2] that keeps the learning less noisy.

for epoch in range(60):
    rng.shuffle(data)
    for start in range(0, len(data), 100): # mini-batches of 100 
        V0 = data[start:start + 100]
    
    # Positive phase: real sightings -> experts
    ph0 = sigmoid(V0 @ W + b_h) # p(expert active / data)
    h0 = sample(ph0)
    
    # Negative phase: ONE Gibbs step-> fantasy sightings
    pv1 = sigmoid(h0 @ W.T + b_v)  # p(spot lit/experts)
    v1 = pv1                       # use probabilities
    ph1 = sigmoid(v1 @ W + b_h)    # p(expert active/fantasy)

    # CD-1 update: (correlations in data) - (correlations in fantasy)
    W += lr * (V0.T @ ph0- v1.T @ ph1) / len(V0)
    b_v += lr * (V0- v1).mean(axis=0)
    b_h += lr * (ph0- ph1).mean(axis=0)

As training proceeds, the model gets better at reproducing the sightings, and the reconstruction error falls before leveling off. It settles a little above zero rather than at zero — that is expected: the injected pixel noise sets a floor no model can reproduce exactly, and on top of that the finite model capacity and the CD-1 approximation keep the error from reaching zero.

What the Experts Learned

Because we now have a real map and many experts, we can look at what each expert learned: its column of weights, drawn as a heatmap over the grid. Each expert became a detector for a different part of the neighborhood: red where it says “the dog is likely here,” blue where it says “not here.” Nobody assigned these regions; the experts divided up the map on their own.

Completing a Partial Sighting

Now the payoff. Suppose we only glimpsed part of a sighting: the right half of the map was blocked from view. We feed the model just the visible half and ask it to reconstruct the rest. From the fragment alone, the model recognizes which region the dog is in and fills in the blob it expects to see there.

Letting the Model Dream

Finally, we can run the machinery in reverse. Instead of feeding in data, we let the model generate sightings entirely on its own, by running Gibbs sampling from a random start: exactly the “fantasy data” the training relied on. The model produces plausible blobs in the regions it learned, showing it has captured the shape of the dog’s habits, not just memorized the training days.

This is the whole article made concrete: a product of simple experts, each learning to watch a different part of the world, trained by comparing real data against fantasy data generated with a single Gibbs step and, once trained, able to both complete what it half-sees and imagine what it has learned.

Conclusion

To recap: a Product of Experts models data by multiplying several simpler distributions instead of averaging them. That single choice — product instead of sum — is what lets a PoE produce sharp, confident distributions, staying certain only where all the experts agree and uncertain wherever they disagree.

Along the way you saw why that power comes at a price. The product has to be re-normalized by the partition function Z, which is intractable to compute directly, and that is what makes training hard. The way around it is to compare the real data against “fantasy data” the model generates itself, using Gibbs sampling and, remarkably, a single Gibbs step (Contrastive Divergence) is enough to make it work. In the final example, all of this came together in a real Restricted Boltzmann Machine that learned the dog’s hiding habits across a whole neighborhood, discovering its favorite regions from noisy past sightings, completing a half-seen sighting, and even dreaming up plausible new ones.

As a next step, try modifying the example yourself: change the dog’s habits, add more hiding regions or more experts, and watch how the experts reorganize to cover them. From there, a natural next stop is running an RBM on a real dataset such as the binarized MNIST digits, where the same recipe learns to recognize and reconstruct handwriting.

Thanks for reading! You can connect with me on LinkedIn and GitHub.

References

[1] G. E. Hinton, Products of Experts. In Proceedings of the 9th International Conference on Artificial Neural Networks (ICANN), vol. 1, pp. 1–6, 1999.

[2] G. E. Hinton, Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation, 14(8):1771–1800, 2002. https://doi.org/10.1162/089976602760128018