KL divergence — why the direction you minimize changes what you get

The gambler models the die with $Q$, a single peak stacked on face 1. The die itself follows $P$, which splits its mass almost evenly between face 1 and face 4 — matching $Q$ at face 1 but leaving face 4 almost untouched. Every roll the gambler bets according to $Q$ instead of $P$, something is lost, but nothing yet turns that loss into a number.

A first guess is to line up $P(x)$ and $Q(x)$ face by face and sum the plain differences. That number treats every face as interchangeable; it never asks what a wrong belief actually costs. The cost shows up in coding. An optimal code built for $Q$ spends about $-\log Q(x)$ bits on outcome $x$, the shortest expected length any code tuned to $Q$ can achieve ^[1][1] Cover & Thomas, Elements of Information Theory — source coding chapter“Unverified model-knowledge reference; verify independently.”.

On the die, this overspend is largest exactly on face 4, the face $P$ favors and $Q$ neglects; face 1 costs little because $Q$ already tracks $P$ there. To turn the per-outcome overspend into one number, average it the way rolls actually happen, expectation under the true distribution the true $P$, not under the gambler's mistaken $Q$.

Summing this weighted excess over all six faces gives one scalar mismatch score for the whole die. But face 1's overspend and some other face's possible undershoot could in principle offset each other, and nothing said so far rules out the total landing below zero, which would mean the wrong code somehow saves bits on average.

Take the excess-bit cost for one face of the die, say face 4: $\log P(4) - \log Q(4)$. That is a single number tied to a single outcome. The die has six faces, and $P$ does not visit them equally — it puts almost half its weight on two distant faces and almost none on the rest. Do you add the six per-face costs straight up, or weight each one by how often that face actually comes up under $P$? Get this wrong and the total either treats a face $P$ never sees as equally important, or lets a face $P$ cares about vanish into the average.

Weight each per-face cost by its true frequency and sum: $\mathbb{E}_P[\log P(x) - \log Q(x)] = \sum_x P(x)\left(\log P(x) - \log Q(x)\right) = \sum_x P(x)\log\frac{P(x)}{Q(x)}$. This sum has a name, the forward KL divergence.

On the die, suppose $P$ puts $0.45$ on face 1 and $0.45$ on face 4, with $0.025$ spread over the remaining four faces. The gambler's $Q$ is unimodal: $0.6$ on face 1, small residual probability elsewhere. The face-4 term is $0.45\log\frac{0.45}{Q(4)}$; if $Q(4)$ is small this single log-ratio is large, and it still gets multiplied by $0.45$ because that is what $P$ actually assigns to face 4. The face-1 term stays small because $Q(1)\approx P(1)$. Adding all six terms produces one scalar, the mismatch cost of using $Q$ on a die genuinely distributed as $P$.

Nothing in that sum treats $P$ and $Q$ interchangeably. Every weight in front of a log-ratio term comes from $P$, never from $Q$; swap which distribution supplies the weights and the total changes for a structural reason, not from rounding. Whether this number can ever come out negative — letting a badly mismatched $Q$ score as though it cost nothing — is still open.

Suppose the gambler's $Q$ happens to overlap one of the die's two heavy faces while missing the other one entirely. At the overlapping face the log-ratio $\log\frac{P(x)}{Q(x)}$ turns negative, since $Q$ now assigns that face even more probability than $P$ does. At the missed face the log-ratio is a large positive number. Could a clever choice of $Q$ push the $P$-weighted sum of these terms below zero, making the mismatch score negative?

Try it directly. Let $P(1)=0.45$, $P(4)=0.45$, and put $Q(1)=0.7$, $Q(4)=0.05$. The face-1 term is $0.45\log\frac{0.45}{0.7}\approx -0.13$, negative as hoped. But the face-4 term is $0.45\log\frac{0.45}{0.05}\approx 0.99$, and it overwhelms the negative one. Shuffle the twelve numbers however you like and the same thing happens: mass $Q$ steals from an underweighted face produces a positive log-ratio large enough to cancel whatever negative term the overlap bought. That is not a coincidence of this particular $Q$ ^[1][1] Cover & Thomas, Elements of Information Theory — source coding chapter“Unverified model-knowledge reference; verify independently.”.

Jensen's inequality is tight exactly when the quantity being averaged, $Q(x)/P(x)$, is constant across every face with $P(x)>0$. Since both $P$ and $Q$ sum to 1 over the same six faces, that constant can only be 1, forcing $Q(x)=P(x)$ everywhere. This is why the two-peak, one-peak setup above never actually reaches zero: $Q$ cannot equal $P$ while restricted to a single mode, so some positive residual is guaranteed no matter how the gambler tunes it.

So $D_{KL}(P\|Q)$ is a legitimate non-negative mismatch score, zero only when the gambler's $Q$ matches $P$ face for face. It is still not a distance in the usual sense: relabel which distribution on this same die gets called $P$ and which gets called $Q$, and the number the formula returns changes, not just cosmetically.

You already have $D_{KL}(P\|Q)$ for the die: true $P$ splits its weight almost evenly across two distant faces, and the gambler's $Q$ piles almost everything onto one face sitting between them. Flip the formula — write $Q$ first, $P$ second — and recompute. A number that behaves like a distance between two objects shouldn't care which one you call first. Swap $P$ and $Q$ everywhere in $D_{KL}(P\|Q)=\sum_x P(x)\log\frac{P(x)}{Q(x)}$ and check whether the result matches what you already found.

It doesn't match. forward KL divergence weights every log-ratio by $P(x)$: the two distant faces the gambler barely touches carry heavy weight, because that's where $P$ lives. reverse KL divergence weights the same kind of ratio by $Q(x)$ instead: the single face the gambler favors carries almost all the weight, because that's where $Q$ lives ^[1][1] Cover & Thomas, Elements of Information Theory — source coding chapter“Unverified model-knowledge reference; verify independently.”. Swapping $P$ and $Q$ doesn't just flip a sign — it moves the averaging measure onto a different set of faces. On a face where $P$ is large and $Q$ is tiny, $D_{KL}(P\|Q)$ pays a heavy $P(x)$-weighted penalty for that mismatch; $D_{KL}(Q\|P)$ barely notices, because $Q(x)$ is tiny there and the term gets almost no weight. The two sums are built from the same log-ratios but averaged under different distributions, so in general $D_{KL}(P\|Q) \neq D_{KL}(Q\|P)$.

Notice what the swap actually cost you: it silently changed which mistakes matter. A modeler who fits by minimizing $D_{KL}(P\|Q)$ is telling the procedure to worry most about faces where the true die rolls often. A modeler who minimizes $D_{KL}(Q\|P)$ is telling it to worry most about faces where its own guess rolls often. Choosing which distribution sits on which side of the bar is a modeling decision with consequences, not free notation.

Force $Q$'s single hump onto face $1$: $Q(1)$ soaks up most of the mass, and $Q(6)$ is left near zero. But $P(6)$ is close to $P(1)$, so the per-outcome excess bit cost term at face $6$, $\log P(6) - \log Q(6)$, explodes as $Q(6) \to 0$, and $D_{KL}(P\|Q)$ takes on a huge value from that one face alone. Put the hump on face $6$ instead and the same disaster happens at face $1$. Neither placement is cheap, because $D_{KL}(P\|Q) = ⁣blurbterm005b0022006500780070006500630074006100740069006f006e002d00500022002c006e0075006c006c005d⁣[\log P(x) - \log Q(x)]$ is an average taken under $P$: any face $P$ visits often but $Q$ nearly ignores gets charged, repeatedly, at whatever rate $P$ actually visits it.

Minimizing $D_{KL}(P\|Q)$ over $Q$ therefore refuses to let $Q(x)$ sit near zero at any face where $P(x)$ is not already near zero, call it zero-avoiding. With $Q$ restricted to one hump, the cheapest single location is the one that keeps both heavy faces at moderate distance rather than starving either one: somewhere between face $1$ and face $6$. The optimal one-peak $Q$ lands in that middle ground, spreading mass toward both true peaks instead of committing to one. This is exactly what maximum likelihood does on real data: fitting $Q$ by maximizing likelihood over samples drawn from $P$ minimizes $D_{KL}(P\|Q)$ up to a constant that does not depend on $Q$, so a unimodal model fit this way to bimodal data blurs between the two modes rather than picking one and abandoning the other.

That coverage is not free, it is just an expense forward KL never bills. To keep both peaks fed, $Q$'s single hump also has to cover the faces between them, faces $2$ through $5$, where $P$ places almost no mass. Nothing in $⁣blurbterm005b0022006500780070006500630074006100740069006f006e002d00500022002c006e0075006c006c005d⁣[\cdot]$ penalizes $Q$ for spending probability where $P$ rarely visits, since $P(x)$ near zero there makes that term's weight near zero too. A direction that instead averages under $Q$ would never let $Q$ pay for outcomes it does not itself predict.

Try repeating the forward-KL fix here: average the log-ratio under $P$, the true posterior over which die face actually generated the data. That posterior is exactly the object variational inference cannot compute — its normalizing constant sums over every latent configuration, the reason the gambler needs an approximation $Q$ in the first place. Averaging under $P$ requires already knowing $P$. The only distribution the gambler can sample from and average under is $Q$ itself.

Swap which distribution does the averaging: $D_{KL}(Q\|P) = \sum_x Q(x)\log\frac{Q(x)}{P(x)}$, the reverse KL divergence. Every term is now weighted by $Q(x)$, so a face where $Q(x)=0$ contributes nothing to the sum no matter how large $P(x)$ is there. The gambler's single-peak $Q$ is free to ignore one of the two true peaks entirely, as long as it puts all its remaining mass on the other. Minimizing $D_{KL}(Q\|P)$ over $Q$ means picking whichever peak makes the log-ratio smallest at the mass $Q$ actually assigns — collapsing onto one peak of $P$, the cheapest one to sit on, and treating the second peak as though it does not exist.

This is mode-seeking behavior, and it is not a modeling preference — it is forced by tractability. Variational inference optimizes a bound built from expectations under $Q$ alone, never under the true posterior $P$, which is exactly why the direction flips ^[2][2] Blei, Kucukelbir & McAuliffe, Variational Inference: A Review for Statisticians“Unverified model-knowledge reference; verify independently.”. The same asymmetric formula, the same two-peaked die, produces opposite failures depending on which distribution does the averaging: forward KL spreads $Q$ across both peaks and pays for faces it barely visits, while reverse KL refuses to pay for any face it does not already own and starves the second peak completely.

Maximum likelihood and variational inference therefore inherit opposite failure modes from the same die and the same formula — one blurs across both peaks, the other truncates to one — and telling which direction produced a given fitted $Q$ from its shape alone is the open diagnostic problem.

Hand someone a fitted $Q$ for the die and hide the training procedure. All they see is a histogram: either mass smeared across faces $P$ barely visits, or a tight spike on one face while a second, equally plausible face gets nothing. Which loss produced which picture?

Run the causal chain from each direction backward. Forward $D_{KL}(P\|Q)$ averages the log-ratio under $P$, so it charges $Q$ for every face $P$ actually visits, including both of the die's true peaks. A single-peak $Q$ minimizing that cost has to compromise, sliding its one mode toward the midpoint of the two peaks and picking up nonzero probability on the faces in between. The symptom is diffusion: mass leaking onto outcomes $P$ considers implausible. That is the signature of maximum likelihood, which is exactly a forward-KL fit to the data-generating $P$ ^[1][1] Cover & Thomas, Elements of Information Theory — source coding chapter“Unverified model-knowledge reference; verify independently.”.

Reverse $D_{KL}(Q\|P)$ averages the same log-ratio under $Q$ instead, so any face $Q$ never visits contributes nothing to the cost regardless of how much probability $P$ puts there. A single-peak $Q$ minimizing that cost has no incentive to compromise; it commits entirely to whichever peak is cheaper and reports zero probability for the other, real, peak. The symptom is confident omission, not diffusion. That is the signature of variational inference, forced into this direction because the true posterior $P$ is intractable to average under ^[2][2] Blei, Kucukelbir & McAuliffe, Variational Inference: A Review for Statisticians“Unverified model-knowledge reference; verify independently.”.

So the diagnostic reads backward from the shape of $Q$: spread mass on low-probability outcomes implicates an average taken under $P$; a confident peak with a starved alternative implicates an average taken under $Q$. The direction of averaging is not a technicality inside the formula, it is the whole difference between a model that hedges across every mode of the truth and one that bets everything on the cheapest mode it can find. Two log-ratios, two directions of averaging, two failure modes visible in any KL-trained model without ever seeing its loss function.