cross-entropy loss — from information theory to gradient flow

Imagine sending a stream of animal classifications for images where every image is guaranteed to be a cat, so the true distribution has $P(\text{cat}) = 1$, $P(\text{dog}) = 0$, and $P(\text{bird}) = 0$. If we assign a fixed naive 2-bit code to each class—say 00 for cat, 01 for dog, and 10 for bird—every message costs exactly $2$ bits ^[1][1] Shannon (1948) A Mathematical Theory of Communication“Model-knowledge reference; verify independently.”. But because dog and bird never occur, sending 00 every single time is completely redundant; zero bits of information are conveyed because there is no uncertainty. Even in a more balanced case where cat occurs $80\%$ of the time while dog and bird occur $10\%$ each, allocating equal 2-bit codes to all three classes wastes transmission capacity on frequent events while under-utilizing rare ones.

To minimize the expected message length, information theory assigns variable-length binary codes based on event probability. An outcome $x$ occurring with true probability $P(x)$ carries an information content—or surprise—measured in bits by $L(x) = -\log_2 P(x)$. When an event is certain ($P(x) = 1$), its surprise is $-\log_2 1 = 0$ bits, requiring no code space. When an outcome is less frequent, such as $P(x) = 0.125$, its surprise is $-\log_2 0.125 = 3$ bits, justifying a longer binary code.

Taking the expectation of this optimal code length across all possible outcomes sampled from true distribution establishes the theoretical lower bound on average message length, defined as Shannon entropy.

For our deterministic target distribution $P = [1, 0, 0]$, calculating entropy yields $H(P) = -1 \log_2 1 - 0 - 0 = 0$ bits. Perfect certainty requires zero bits on average to transmit. If $P$ were instead a uniform distribution over four classes, $P = [0.25, 0.25, 0.25, 0.25]$, the entropy would reach $H(P) = -4 \times (0.25 \log_2 0.25) = 2$ bits, requiring $2$ bits per outcome. Shannon's source coding theorem guarantees that no lossless compression scheme can achieve an average code length shorter than $H(P)$ when messages are generated by distribution $P$ ^[1][1] Shannon (1948) A Mathematical Theory of Communication“Model-knowledge reference; verify independently.”.

This establishes the absolute lower bound when our compression scheme knows the true distribution $P(x)$ perfectly. In machine learning, however, the target distribution $P(x)$ is hidden, and our model instead outputs an estimated distribution $Q(x)$. What average bit penalty do we pay when we construct our coding scheme using estimated model probabilities $Q(x)$ instead of the true distribution $P(x)$?

Consider classifying an image where the true target is cat, represented by target distribution true distribution $= [1, 0, 0]$. Suppose a neural network outputs predicted probabilities predicted distribution $= [0.1, 0.7, 0.2]$, assigning $70\%$ probability to dog and only $10\%$ to the true cat label. If we evaluate this mistake using a naive linear difference like $1 - q_1 = 0.9$, the metric penalizes all probability deficits uniformly. Under a linear metric, losing $0.1$ probability when moving from $q_1 = 0.9$ to $0.8$ incurs the exact same penalty as dropping from $q_1 = 0.1$ to $0.0$. Yet these two failures represent fundamentally different model behaviors. Assigning $q_1 \approx 0$ to an event that actually occurs means the network considers the true outcome virtually impossible. In information theory, representing an event assigned probability $q_1$ requires a code word of length $-\log Q(x)$ ^[1][1] Shannon (1948) A Mathematical Theory of Communication“Model-knowledge reference; verify independently.”. As $q_1$ approaches zero, the required code length approaches infinity, a catastrophe that linear metrics completely fail to capture.

To quantify the expected penalty incurred by assuming estimated distribution predicted distribution instead of true distribution true distribution, we compute the expected code length across all possible outcomes under $P$. This expectation defines the cross-entropy ^[1][1] Shannon (1948) A Mathematical Theory of Communication“Model-knowledge reference; verify independently.”. For a target distribution $P = [1, 0, 0]$ and model output $Q = [q_1, q_2, q_3]$, the sum over all classes collapses because non-target classes have $P(x) = 0$:

$H(P, Q) = -1 \cdot \log q_1 - 0 \cdot \log q_2 - 0 \cdot \log q_3 = -\log q_1$

When the model assigns full confidence to the correct class ($q_1 = 1$), the loss vanishes to $-\log(1) = 0$, indicating that no excess code space is wasted. However, when the model makes a confident wrong prediction like $Q = [0.1, 0.7, 0.2]$, the target probability $q_1 = 0.1$ yields a cross-entropy loss of $-\log(0.1) \approx 2.30$. If the model becomes even more confident in its mistake and drops $q_1$ to $0.01$, the loss doubles to $-\log(0.01) \approx 4.61$. In the extreme limit as $q_1 \to 0$, the loss grows without bound towards infinity:

$\lim_{q_1 \to 0} H(P, Q) = \lim_{q_1 \to 0} (-\log q_1) = \infty$

By penalizing probability deficits logarithmically rather than linearly, cross-entropy loss generates extremely steep gradients precisely when the model is confidently wrong. This property prevents the optimization process from stalling on severe misclassifications.

Evaluating $H(P, Q)$ penalizes missing target probability with infinite severity, but does minimizing cross-entropy directly minimize an explicit distance between probability distributions $P$ and $Q$?

When optimizing model parameters $\theta$ to classify our cat image with ground-truth target $P = [1, 0, 0]$ against predicted probabilities $Q = [q_1, q_2, q_3]$, minimizing the cross-entropy loss $H(P, Q) = -1 \log q_1$ appears on the surface to depend on the intrinsic structure of $P$. Because cross-entropy measures the total expected code length paid when events from $P$ are encoded using $Q$, one might worry that loss updates could be confounded by variations in target uncertainty. If cross-entropy combines both target complexity and distribution mismatch into a single scalar value, could gradient descent inadvertently attempt to alter the underlying target distribution's baseline entropy rather than focusing purely on model alignment? ^[2][2] Kullback & Leibler (1951) On Information and Sufficiency“Model-knowledge reference; verify independently.”

To test whether cross-entropy cleanly separates prediction error from target uncertainty, we decompose the total cross-entropy $H(P, Q)$ into two distinct terms: the baseline theoretical minimum code length given by target Shannon entropy $H(P)$, and the extra bit penalty incurred by using model probabilities $Q$ instead of true probabilities $P$. This excess penalty is called the forward KL divergence $D_{KL}(P \| Q)$.

We write the forward KL divergence by taking the expectation over target distribution $P$ of the logarithmic difference between true probabilities $P(x)$ and predicted probabilities $Q(x)$:

Using standard logarithm rules to expand the log quotient $\log \frac{P(x_i)}{Q(x_i)} = \log P(x_i) - \log Q(x_i)$, we split the summation into two separate sums:

We recognize the first term $-\sum_{i=1}^3 P(x_i) \log P(x_i)$ as the negative target entropy $-H(P)$, and the second term $-\sum_{i=1}^3 P(x_i) \log Q(x_i)$ as the cross-entropy $H(P, Q)$. Substituting these definitions yields the fundamental decomposition formula:

This decomposition makes the distinction between target uncertainty and model miscalibration explicit. For our cat classification target $P = [1, 0, 0]$, target entropy is $H(P) = -1 \log(1) - 0 - 0 = 0$. Even in settings with soft targets where $H(P) > 0$, the target distribution $P$ is fixed by the dataset and contains no trainable model parameters $\theta$. When taking the derivative of cross-entropy with respect to network weights $\theta$, the target entropy derivative $\nabla_\theta H(P)$ vanishes identically:

Because $\nabla_\theta H(P) = 0$, updating network parameters $\theta$ along the negative gradient of cross-entropy $-\nabla_\theta H(P, Q)$ is mathematically identical to updating parameters along $-\nabla_\theta D_{KL}(P \| Q)$. The optimization algorithm cannot modify the dataset's intrinsic entropy; every parameter update works exclusively to minimize the statistical discrepancy between predicted probabilities $Q$ and target distribution $P$.

Cross-entropy isolates distribution mismatch mathematically, but how does this log-based objective behave when optimized via gradient descent through continuous logit activations?

In our cat classification task, the model converts raw, unnormalized logits $z = [z_1, z_2, z_3]$ into predicted probabilities $Q = [q_1, q_2, q_3]$ using the softmax function. Because softmax normalizes every component by the sum of exponentials $\sum_k e^{z_k}$, evaluating its partial derivatives introduces quotient terms and coupled cross-terms. The derivative of every predicted probability $q_j$ with respect to every input logit $z_i$ yields a complex Jacobian matrix containing terms like $q_i(1 - q_i)$ and $-q_i q_j$. Navigating this interconnected quotient structure during backpropagation threatens severe computational overhead and potential gradient vanishing.

To compute the loss gradient with respect to logit $z_i$, we apply the multivariable chain rule across all output probabilities: $\frac{\partial L}{\partial z_i} = \sum_j \frac{\partial L}{\partial q_j} \frac{\partial q_j}{\partial z_i}$. Differentiating loss $L$ gives $\frac{\partial L}{\partial q_j} = -\frac{p_j}{q_j}$. For the softmax output, the Jacobian entry $\frac{\partial q_j}{\partial z_i}$ equals $q_i(1 - q_i)$ when $j = i$ and $-q_j q_i$ when $j \neq i$. Substituting these components into the chain rule expansion gives:

$\frac{\partial L}{\partial z_i} = -\frac{p_i}{q_i} \left( q_i(1 - q_i) \right) + \sum_{j \neq i} -\frac{p_j}{q_j} \left( -q_j q_i \right)$

Notice the algebraic simplification: the $q_i$ denominator in $-\frac{p_i}{q_i}$ cancels the $q_i$ factor in $q_i(1 - q_i)$, while $q_j$ in $-\frac{p_j}{q_j}$ cancels $q_j$ inside $-q_j q_i$. Simplifying the remaining terms yields $-p_i(1 - q_i) + \sum_{j \neq i} p_j q_i = -p_i + p_i q_i + \sum_{j \neq i} p_j q_i = -p_i + q_i \sum_j p_j$. Because $P$ is a valid probability distribution, target probabilities sum to $1$ ($\sum_j p_j = 1$). The expression collapses completely to $\frac{\partial L}{\partial z_i} = q_i - p_i$ ^[1][1] Shannon (1948) A Mathematical Theory of Communication“Model-knowledge reference; verify independently.”. In vector form, $\nabla_z L = Q - P$.

This exact cancellation between the softmax exponential and cross-entropy logarithm leaves a purely linear residual error vector $Q - P$. When the model is confidently wrong ($q_1 \approx 0$ for target $p_1 = 1$), the gradient magnitude $|q_1 - p_1| \approx 1$ guarantees strong, constant parameter updates. But what happens if we pair softmax outputs with an alternative loss function like Mean Squared Error instead?

Suppose our network evaluates a cat image ($P = [1, 0, 0]$) and outputs a confidently misclassified prediction $Q = [q_1, q_2, q_3] = [0.001, 0.998, 0.001]$. Measuring probability error with Mean Squared Error Mean Squared Error loss yields a loss penalty $(q_1 - 1)^2 \approx 1$, near its maximum theoretical bound ^[3][3] Golik et al. (2013) Cross-Entropy vs Squared Error Training Samples“Model-knowledge reference; verify independently.”. Intuition suggests that such a severe error should trigger steep gradient updates to correct the model parameters model parameters. Instead, backpropagation stalls completely, leaving the network trapped in its incorrect state.

To understand why parameter updates freeze, trace how the loss gradient flows back through the network to the unnormalized logit $z_1$ logit. Under softmax activation, probability $q_1 = \frac{e^{z_1}}{\sum_k e^{z_k}}$ depends on all class logits. The partial derivative of $q_1$ with respect to logit $z_1$ is $\frac{\partial q_1}{\partial z_1} = q_1(1 - q_1)$. By the chain rule, the gradient of any loss function $L$ with respect to logit $z_1$ is $\frac{\partial L}{\partial z_1} = \frac{\partial L}{\partial q_1} \frac{\partial q_1}{\partial z_1}$.

For Mean Squared Error, taking the derivative with respect to predicted probability $q_1$ gives $\frac{\partial L_{\text{MSE}}}{\partial q_1} = q_1 - p_1 = q_1 - 1$. Substituting this into the chain rule yields the explicit logit derivative:

When the model is confidently wrong ($q_1 \approx 0$), the factor $q_1$ in the softmax derivative vanishes to zero, driving the entire product $\frac{\partial L_{\text{MSE}}}{\partial z_1} \to 0$. Although the loss derivative $q_1 - 1 \approx -1$ indicates a massive prediction error, it is scaled down by the vanishing softmax term $q_1(1 - q_1) \approx 0.000999$. Consequently, the gradient becomes negligible, parameter updates cease, and learning freezes.

Cross-entropy loss avoids this saturation through exact derivative cancellation. The cross-entropy loss derivative with respect to probability $q_1$ is $\frac{\partial L}{\partial q_1} = -\frac{1}{q_1}$. Multiplying by the softmax derivative gives:

The factor $q_1$ in the denominator of the cross-entropy derivative perfectly cancels the vanishing term $q_1$ in the softmax derivative. Even when $q_1 = 0.001$, cross-entropy maintains a robust gradient of $\frac{\partial L}{\partial z_1} = 0.001 - 1 = -0.999$, ensuring rapid parameter updates until the prediction matches the target.

Cross-entropy eliminates gradient saturation, but its drive to reach zero loss forces target logit $z_1 \to \infty$, causing model overconfidence.

With a hard one-hot target $P = [1, 0, 0]$ for classifying our cat image, cross-entropy loss $L = -\log q_1$ reaches its theoretical minimum of zero only when the predicted probability $q_1 = 1$. Expressed in terms of unnormalized logits logit, predicted probability $q_1 = \frac{e^{z_1}}{e^{z_1} + e^{z_2} + e^{z_3}}$. For $q_1$ to equal $1$, the logit difference $z_1 - z_j$ for non-target classes $j \in \{2, 3\}$ must approach infinity. As gradient descent continuously updates model parameters model parameters, the optimization algorithm relentlessly increases weight magnitudes to drive target logit $z_1 \to \infty$ relative to $z_2$ and $z_3$. This logit explosion causes extreme probability overconfidence: the network outputs probabilities arbitrarily close to $1.0$ on training samples, destroying probability calibration and making hidden representations rigid and fragile to noise ^[4][4] Szegedy et al. (2016) Rethinking the Inception Architecture for Computer Vision“Model-knowledge reference; verify independently.”.

Label smoothing resolves logit explosion by altering target distribution $P$ before computing cross-entropy loss cross-entropy. For $K = 3$ total classes and smoothing factor $\epsilon = 0.1$, the original one-hot vector $P = [1, 0, 0]$ softens into target distribution $P_\epsilon = [0.933, 0.033, 0.033]$. The cross-entropy loss $H(P_\epsilon, Q)$ reaches its absolute minimum when predicted probability vector $Q = [q_1, q_2, q_3]$ perfectly matches $P_\epsilon$, setting $q_1 = 0.933$ and $q_2 = q_3 = 0.033$.

To produce prediction $q_1 = 0.933$ under softmax normalization, target logit $z_1$ no longer needs to approach infinity. Instead, the optimal logit difference $z_1 - z_2$ is finite: $\ln(0.933 / 0.033) \approx 3.342$. Because the loss function penalizes predictions that overestimate target certainty beyond $0.933$, gradient descent stops expanding weight magnitudes once logit differences reach this finite threshold.

This completes our throughline from information theory to backpropagation dynamics. Measuring categorical mismatch using cross-entropy loss guarantees non-saturating linear gradients $q_i - p_i$ that preserve steep updates when predictions are wrong, avoiding the vanishing gradient failure of quadratic loss. Incorporating label smoothing regularizes optimal logit magnitudes, preventing infinite weight growth while preserving clear margin boundaries between classes.