Cross-Entropy Loss Calculator

Cross-entropy loss is the negative logarithm of the probability a model assigns to the correct answer, averaged over the samples, so it measures how surprised the model is by the truth. This cross entropy loss calculator returns that loss in nats and bits, the perplexity and the true-class probability from predicted probabilities, raw logits or binary labels.

What is cross-entropy loss?

A classifier does not just pick a class. It spreads probability over every class, and cross-entropy scores it on one thing: how much of that probability landed on the class that turned out to be right. Put 99% there and the model is barely surprised, so the loss is close to 0. Put 1% there and the model is badly surprised, so the loss is large.

The surprise is measured with a logarithm, which is what gives the loss its character. Being confidently wrong costs far more than being unsure, and the cost has no ceiling as the probability of the true class approaches 0. The curve in the tool shows that shape, with your own prediction marked on it.

In binary classification the same quantity is called log loss or binary cross-entropy. Formally it is an expected value: the average of −ln p taken under the target distribution.

What is the cross-entropy loss formula?

\(p_i\) is the predicted probability of class \(i\), \(y_i\) is the target probability of that class, and \(\ln\) is the natural logarithm. With an ordinary hard label, \(y\) is 1 for the true class \(t\) and 0 elsewhere, so the sum collapses to a single term:

Networks output raw scores called logits, \(z_i\), which softmax turns into probabilities. Substituting softmax into the line above gives the form that frameworks actually compute:

For binary labels \(y_n \in \{0, 1\}\) and predicted probabilities \(p_n\) of class 1, the loss averaged over \(N\) samples is:

Worked example: logits 2, 1, 0.1

Take a three-class model that outputs the logits 2, 1 and 0.1 for a sample whose true class is 0. These are the default inputs of the calculator. The exponentials are e² = 7.389056, e¹ = 2.718282 and e^0.1 = 1.105171, which add up to 11.212509.

Dividing each by the sum gives the softmax probabilities 0.659001, 0.242433 and 0.098566. The true class received 0.659001, so the loss is −ln(0.659001) = 0.417030 nats. That is 0.417030 / 0.693147 = 0.601647 bits, and the perplexity is e^0.417030 = 1.517448.

A uniform guess over three classes would score ln 3 = 1.098612, so this prediction is well ahead of chance.

How does the loss change with the probability of the true class?

Every row below is −ln p for a single sample. The perplexity column is 1 / p, the number of equally likely options the model behaves as if it were choosing between.

Moving from 0.9 to 0.99 saves less than 0.1 nats. Dropping from 0.1 to 0.01 adds 2.3. Because the average is dominated by its worst terms, a handful of confidently wrong predictions can outweigh thousands of good ones.

What is a good cross-entropy loss?

There is no fixed threshold, because the scale depends on the number of classes. The useful reference is the loss of a model that knows nothing and predicts every one of K balanced classes with probability 1/K, which is ln K: 0.6931 for 2 classes, 2.3026 for 10, 4.6052 for 100 and 6.9078 for 1,000.

A freshly initialised network should start near that number. If the first logged loss is far above ln K, suspect the initialisation or the labels. If the loss stays at ln K, the model is not learning yet, and the learning rate schedule is the first thing to check. When classes are imbalanced, the fairer baseline is the entropy of the class frequencies, which is lower than ln K.

Nats, bits and perplexity

The unit follows the base of the logarithm. The natural log gives nats, which is what PyTorch, TensorFlow and scikit-learn report. Dividing by ln 2 = 0.693147 converts to bits, the unit of information theory and of the entropy used in information gain for decision trees.

Perplexity is e raised to the loss in nats. Language models report it because it reads as a branching factor: a mean cross-entropy of 2.995732 nats per token is a perplexity of 20, meaning the model is as uncertain as if it were choosing uniformly among 20 tokens at each step.

Where is cross-entropy used in machine learning?

• Multi-class classifiers. A softmax output layer trained with cross-entropy is the standard setup for image and text classification. The gradient with respect to each logit is simply \(p_i - y_i\), which is why the pairing trains so smoothly.
• Binary classifiers and logistic regression. The output passes through a sigmoid and is scored with binary cross-entropy. Fitting logistic regression by maximum likelihood is the same as minimising log loss.
• Language models. Next-token prediction is classification over the vocabulary, and the training loss is the mean cross-entropy per token.
• Soft targets. Label smoothing and knowledge distillation replace the one-hot target with a distribution. With a target of 0.5, 0.5, 0 and predictions of 0.7, 0.2, 0.1 the loss is 0.983056, and the lowest value it can reach is the entropy of the target, 0.693147, not 0.
• Model evaluation. Log loss judges the probabilities themselves, while accuracy and the F1 score judge only the decisions made after thresholding. Two models with identical accuracy can have very different log loss.

Does PyTorch CrossEntropyLoss expect logits or probabilities?

Logits. torch.nn.CrossEntropyLoss and torch.nn.functional.cross_entropy take raw, unnormalised scores and apply log-softmax internally. Applying softmax yourself first is a bug that raises no error: the function treats your probabilities as logits and applies softmax a second time.

In the worked example, the double softmax reports 0.802121 instead of 0.417030. Worse, the inputs are now confined to the range 0 to 1, so with three classes the loss can never fall below ln(1 + 2/e) = 0.551445 and the gradients shrink. Training still runs, just badly.

The binary losses have the same split. BCEWithLogitsLoss takes logits and applies the sigmoid itself in a numerically stable way, while BCELoss expects probabilities that have already been through a sigmoid. Use the first unless you have a reason not to.

Common mistakes

• Softmax before a loss that expects logits. See the section above. The mirror-image error is feeding logits to a loss that expects probabilities.
• Using the wrong logarithm. Log base 10 or base 2 gives a different number. Frameworks use the natural log.
• Off-by-one class indices. Targets are counted from 0 in PyTorch, Keras and this calculator. Class 3 of 3 does not exist.
• Letting a probability hit exactly 0. ln 0 is minus infinity. Libraries clip probabilities to a tiny positive value, and this tool clips at 1e-15 and tells you when it did.
• Comparing losses across different class counts. A loss of 1.5 is poor for 3 classes and excellent for 1,000. Compare against ln K.
• Mixing a sum with a mean. Frameworks average over the batch by default. A summed loss grows with batch size and is not comparable.

How do you calculate cross-entropy loss in Python?

The NumPy version below uses the same max-subtraction trick as the calculator, and the PyTorch version passes the raw logits straight to the loss. Both reproduce the worked example.

# NumPy: stable log-softmax, then pick the target class
import numpy as np

z = np.array([2.0, 1.0, 0.1])   # logits
target = 0

shifted = z - z.max()
log_p = shifted - np.log(np.exp(shifted).sum())
loss = -log_p[target]

print(np.round(np.exp(log_p), 6))    # [0.659001 0.242433 0.098566]
print(f"{loss:.6f}")                 # 0.417030
print(f"{loss / np.log(2):.6f}")     # 0.601647  (bits)
print(f"{np.exp(loss):.6f}")         # 1.517448  (perplexity)

# PyTorch: pass raw logits, not softmax output
import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 1.0, 0.1]])   # shape (batch, classes)
target = torch.tensor([0])                 # class index per sample

loss = F.cross_entropy(logits, target)
print(f"{loss.item():.6f}")                # 0.417030

The Keras equivalent is tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), called with the integer labels first and the logits second.

Frequently asked questions

Is cross-entropy loss the same as log loss?

Yes. Log loss is the usual name for cross-entropy when it is used to evaluate predicted probabilities, especially in binary classification. With hard labels both reduce to minus the natural log of the probability given to the true class, averaged over the samples.

Can cross-entropy loss be greater than 1?

Yes. The loss has no upper limit. A probability of 0.1 on the true class costs 2.302585 nats and a probability of 0.01 costs 4.605170. Only a loss of 0 is special: it means the model gave the true class a probability of 1.

What is a good value for cross-entropy loss?

Lower is better, and the reference point is ln K for K balanced classes: 0.6931 for 2 classes and 2.3026 for 10. A loss below that beats uniform guessing. How far below is achievable depends on how noisy the labels are, so compare models on the same data instead of chasing a fixed number.

Why is my loss stuck at 0.693?

0.693 is ln 2, the loss of a binary classifier that predicts 0.5 for every sample. A loss that sits there means the model has not learned anything yet. Common causes are a learning rate that is too high or too low, inputs that are not normalised, or labels that do not line up with the inputs.

Related calculators

Cross-entropy sits at the end of a pipeline: a network produces logits, an activation turns them into probabilities, and the loss scores those probabilities. The other tools for that pipeline are collected on the deep learning calculators page.

More tools that pair with this one:

• Turn a vector of logits into class probabilities
• Map a single logit to a probability with the sigmoid
• Entropy and information gain for a decision tree split
• Precision, recall and F1 after thresholding