Entropy and cross-entropy: why your loss function is an information measure

Every time a language model predicts the next token, it makes a bet about the future. If it assigns a high probability to the token that actually appears, the bet was good. If it assigns a tiny probability, the model pays a penalty.

That penalty is not arbitrary. It comes directly from information theory.

The cross-entropy loss used to train language models measures how many units of information the model needs to account for the data it observes. Entropy describes uncertainty, cross-entropy measures the cost of predicting with a particular probability distribution, and Kullback–Leibler (KL) divergence measures the extra cost of using the wrong distribution.

These are not merely three related mathematical concepts. They are three perspectives on the same fundamental problem: how efficiently can a probability model describe reality?

Understanding this connection explains why:

• Cross-entropy is such a natural objective for classification and language modeling.

• Confidently wrong predictions receive disproportionately large penalties.

• A model’s validation loss can be interpreted as a compression score.

• Perplexity is an exponentiated form of cross-entropy.

• Two language models can report different perplexities partly because they use different tokenizers.

• Bits per byte is increasingly useful for comparing language models.

We’ll derive the essential relationships, calculate all three information measures on real text with Python, build a simple predictive text model, and connect its compression score to measurements from modern large language models.

1. Entropy starts with uncertainty, not machine learning

Imagine building a system that predicts the next event from a small collection of possibilities.

Consider an online store where the next order belongs to one of four product categories:

We know which events are possible and how likely each one is. But we do not know which particular category the next order will belong to.

How much uncertainty remains?

A distribution that assigns equal probabilities to every category is harder to predict than one that strongly favors a single category. If every future order belonged to electronics, there would be no category uncertainty at all.

Entropy captures this intuition mathematically.

For a discrete random variable X with possible outcomes x, Shannon entropy is:

H(X) = −Σ p(x) log₂ p(x)

The sum runs over every possible outcome.

The base-2 logarithm makes the unit bits. The convention is that an outcome with probability zero contributes zero to the entropy sum.

Why the logarithm?

Information theory assigns an individual outcome a quantity called surprisal:

I(x) = −log₂ p(x)

A common event carries less information because it is less surprising. A rare event carries more.

For example:

If an outcome has probability 1/1024, observing it gives us 10 bits of information under that probability model.

The logarithm has another important property: it makes information additive.

If two events are independent, their joint probability is the product of their probabilities. Taking the negative logarithm turns that product into a sum.

I(a, b) = I(a) + I(b)

That is precisely the behavior we want when measuring the information required to describe a sequence of independent events.

Calculate entropy by hand

Returning to our store categories:

H(X) = −(0.5 log₂ 0.5 + 0.25 log₂ 0.25 + 0.125 log₂ 0.125 + 0.125 log₂ 0.125)

Substituting the logarithms:

H(X) = 0.5 × 1 + 0.25 × 2 + 0.125 × 3 + 0.125 × 3

H(X) = 1.75 bits

This is the expected surprisal of a category drawn from that distribution.

A uniform distribution over four categories would have entropy 2 bits. Our store is somewhat more predictable because electronics orders are more common.

There is also a practical interpretation: entropy gives the theoretical average code length for representing outcomes from a known distribution, in the asymptotic limit of efficient lossless coding. A simple prefix code for our four categories can achieve exactly 1.75 bits per category.

The connection between probability and coding is the first clue that a machine-learning loss function might also be a compression measure.

2. Cross-entropy: what happens when the model believes the wrong probabilities?

Entropy assumes we know the true probability distribution.

In machine learning, we generally do not.

Instead, there are two distributions:

• P: The true distribution generating the observations.

• Q: The distribution predicted by our model.

The model might believe electronics orders are less likely than they really are. It might overestimate books or home goods.

We need a way to measure how expensive those mistaken beliefs are.

That quantity is cross-entropy:

H(P, Q) = −Σ p(x) log₂ q(x)

Notice the difference from ordinary entropy.

Entropy uses the true probability p(x) inside the logarithm. Cross-entropy uses the model’s predicted probability q(x).

The observations still come from P, but the cost of describing them is determined by Q.

A numerical example

Suppose the true category distribution and our model’s predictions are:

The model predicts a uniform distribution. It has failed to learn which categories are more common.

For this uniform Q, every event costs 2 bits, so:

H(P, Q) = 2 bits

But the true entropy was only 1.75 bits.

The model therefore pays an additional 0.25 bits per event because its probability estimates are inaccurate.

This extra cost has a name: KL divergence.

3. KL divergence: the price of a mistaken belief

Kullback–Leibler divergence quantifies the difference between a true distribution P and an approximate distribution Q:

D_KL(P || Q) = Σ p(x) log₂(p(x) / q(x))

Unlike an ordinary geometric distance, KL divergence is not symmetric. In general:

D_KL(P || Q) ≠ D_KL(Q || P)

And if P assigns positive probability to an outcome for which Q assigns zero probability, the divergence is infinite.

The crucial identity is:

H(P, Q) = H(P) + D_KL(P || Q)

In words:

Cross-entropy = irreducible uncertainty + penalty for model mismatch

For our store example:

• Entropy: 1.75 bits.

• Cross-entropy: 2.00 bits.

• KL divergence: 0.25 bits.

The model cannot eliminate uncertainty inherent in the true distribution. But it can eliminate the additional uncertainty created by its inaccurate predictions, provided its model family is expressive enough.

KL divergence is always nonnegative, so cross-entropy is never smaller than the true entropy when both are defined for the same distribution and probability space.

This is why minimizing cross-entropy makes theoretical sense: it pushes the model’s predicted distribution toward the distribution of the data.

Verify the identity in Python

We can check the result without NumPy, PyTorch, or any external dependencies.

import math


def entropy(p):
    return -sum(x * math.log2(x) for x in p if x > 0)


def cross_entropy(p, q):
    total = 0.0
    for px, qx in zip(p, q, strict=True):
        if px > 0:
            if qx <= 0:
                return math.inf
            total -= px * math.log2(qx)
    return total


def kl_divergence(p, q):
    total = 0.0
    for px, qx in zip(p, q, strict=True):
        if px > 0:
            if qx <= 0:
                return math.inf
            total += px * math.log2(px / qx)
    return total


p = [0.5, 0.25, 0.125, 0.125]
q = [0.25, 0.25, 0.25, 0.25]

h = entropy(p)
ce = cross_entropy(p, q)
kl = kl_divergence(p, q)

print(f"Entropy:       {h:.4f} bits")
print(f"Cross-entropy: {ce:.4f} bits")
print(f"KL divergence: {kl:.4f} bits")

assert math.isclose(ce, h + kl)

Output:

Entropy:       1.7500 bits
Cross-entropy: 2.0000 bits
KL divergence: 0.2500 bits

One detail matters when interpreting these results: KL divergence is measured in bits only because we used base-2 logarithms. Using natural logarithms produces nats instead.

Both conventions describe the same underlying information quantity.

4. Why cross-entropy becomes the loss function in machine learning

The formulas so far describe entire probability distributions. But a neural network usually trains on individual examples.

How do we get from information theory to the familiar classification loss?

Suppose a classifier predicts probabilities over three product categories:

• Electronics: 0.70

• Clothing: 0.20

• Books: 0.10

The actual category is electronics.

Representing the observed label as a one-hot distribution means the true class receives probability 1 and the other classes receive probability 0.

When we substitute this one-hot target into cross-entropy, every term disappears except the term for the correct class.

The loss becomes:

Loss = −ln(q(correct class))

Here we use the natural logarithm, so the loss is measured in nats rather than bits.

For the correct electronics prediction:

Loss = −ln(0.70) ≈ 0.357

If the model had predicted only 0.01 probability for the actual class:

Loss = −ln(0.01) ≈ 4.605

That is a much larger penalty.

Why confidently wrong predictions hurt so much

Cross-entropy does not merely check whether the highest-probability class matches the label.

It evaluates the probability assigned to the actual outcome.

This distinction is fundamental:

• A model assigning 0.51 probability to the correct class has a higher loss than one assigning 0.95.

• A model assigning 0.001 probability to the correct class receives a severe penalty, even if its predicted class is wrong by only a small margin.

• A model that consistently assigns good probabilities can have a better cross-entropy score than another model with similar classification accuracy.

This makes cross-entropy a proper scoring rule: in expectation, when the true distribution is P, reporting P minimizes the score. A model is rewarded for calibrated probabilistic predictions, not merely hard decisions.

The connection to maximum likelihood

Consider a dataset containing N independent training examples. Each example has a feature vector and an observed label.

A probabilistic classifier assigns the observed label a conditional probability.

Maximum likelihood estimation selects model parameters that maximize the product of those probabilities:

Likelihood = Π q(yᵢ | xᵢ)

Products of many small probabilities quickly become numerically inconvenient. Taking a logarithm turns the product into a sum.

Maximizing log-likelihood is equivalent to minimizing negative log-likelihood:

NLL = −Σ ln q(yᵢ | xᵢ)

Divide by the number of examples and we obtain average cross-entropy loss for one-hot targets:

Mean loss = −(1/N) Σ ln q(yᵢ | xᵢ)

Therefore, training a probabilistic classifier using ordinary cross-entropy is the same optimization problem as maximum likelihood estimation.

This is the central reason the loss function appears across so many machine-learning architectures. It follows naturally from probability modeling, not from a particular neural-network design.

How this looks in modern PyTorch

PyTorch’s torch.nn.functional.cross_entropy accepts unnormalized logits and class-index targets directly. You should not apply softmax to the logits before passing them to this function; the operation handles the necessary normalization internally using numerically stable computations.

Here is a minimal next-token-style example, using randomly generated logits rather than a trained network:

import torch
import torch.nn.functional as F

torch.manual_seed(42)

vocab_size = 5

token_ids = torch.tensor([[1, 4, 2, 3, 0]])

# The input prefix excludes the final token.
inputs = token_ids[:, :-1]

# Each position predicts the following token.
targets = token_ids[:, 1:]

# Placeholder logits from an untrained model.
logits = torch.randn(
    1, inputs.shape[1], vocab_size,
    requires_grad=True
)

loss = F.cross_entropy(
    logits.reshape(-1, vocab_size),
    targets.reshape(-1)
)

loss.backward()

print(f"Loss: {loss.item():.4f} nats/token")
print(f"Perplexity: {torch.exp(loss).item():.4f}")
print(f"Gradient shape: {tuple(logits.grad.shape)}")

The model would normally produce these logits from embeddings and a neural network. Everything after that point is conceptually the same: compare predicted next-token probabilities with actual next-token identities, accumulate negative log-probabilities, and propagate gradients.

In a real training pipeline, padding positions are usually excluded from the loss with a suitable mask or ignored target index. This prevents artificial padding tokens from distorting the reported average.

5. From classification to language modeling

Language modeling is probabilistic classification repeated over a sequence.

At every position, the model predicts a distribution over its vocabulary.

Suppose the text is:

The train arrived at the station.

After observing the prefix The train arrived at the, the model might assign probabilities such as:

If the next token is station, the model pays:

−ln(0.65) ≈ 0.431 nats

A model that assigns 0.02 probability to station pays:

−ln(0.02) ≈ 3.912 nats

An autoregressive language model performs this prediction for every token, conditioning on its preceding context.

The probability of a complete token sequence is factorized as:

P(x₁, …, xₜ) = Π P(xₜ | x₁, …, xₜ₋₁)

Here the product is understood to run over positions in the sequence, each conditioned on its own preceding tokens.

This factorization does not assume that tokens are independent. Quite the opposite: the model uses the preceding context to predict each successive token.

The negative log-likelihood of a sequence becomes:

Sequence NLL = −Σ ln q(xₜ | preceding tokens)

And average next-token cross-entropy is:

L = Sequence NLL / number of scored tokens

Training a language model to minimize this quantity is equivalent to teaching it to assign higher probabilities to the next tokens that actually appear in text.

The architecture might use attention layers, sparse experts, or another sequence-processing mechanism. The underlying probabilistic objective remains the same.

What is the irreducible loss for language?

Natural language contains uncertainty.

Even with substantial context, multiple continuations may be reasonable. A sentence might continue with station, platform, or a different destination entirely.

An ideal predictor cannot eliminate genuine randomness or information not contained in its context.

The theoretical target is the conditional entropy of the next token given the available history:

H(next token | context)

A learned model’s expected cross-entropy can be understood as this conditional entropy plus an expected KL divergence between the true context-dependent distribution and the model’s predictions.

A caveat matters: training loss on a finite dataset can fall below estimates of irreducible uncertainty through memorization or overfitting. The theoretical decomposition concerns expectations under a specified data-generating distribution, not a guarantee about an observed training run.

That distinction is why held-out validation loss is more informative than training loss when measuring generalization.

6. Perplexity: cross-entropy expressed as effective uncertainty

Language-model papers and evaluation systems frequently report perplexity alongside loss.

Perplexity is simply exponentiated average cross-entropy.

When cross-entropy uses natural logarithms:

Perplexity = exp(L)

When it uses base-2 logarithms:

Perplexity = 2ᴴ

These give the same result when the units are converted consistently.

Interpreting perplexity

Suppose a language model has an average cross-entropy of 2 nats per token.

Its perplexity is:

exp(2) ≈ 7.39

You can think of perplexity as an effective number of equally probable choices at each prediction position.

That interpretation is useful, but approximate: real vocabulary distributions are highly nonuniform and change with context. Perplexity does not mean the model literally considers only 7.39 tokens.

Consider another example:

import math

# Predicted probabilities of the actual next tokens.
actual_token_probabilities = [0.8, 0.1, 0.5, 0.25]

nll = -sum(
    math.log(p) for p in actual_token_probabilities
)

mean_loss = nll / len(actual_token_probabilities)
perplexity = math.exp(mean_loss)
bits_per_token = mean_loss / math.log(2)

print(f"Mean loss:      {mean_loss:.4f} nats/token")
print(f"Perplexity:     {perplexity:.4f}")
print(f"Bits per token: {bits_per_token:.4f}")

Output:

Mean loss:      1.1513 nats/token
Perplexity:     3.1623
Bits per token: 1.6610

The model assigned varying probabilities to four observed next tokens. Perplexity summarizes their average negative log-likelihood in a different scale.

Why perplexity comparisons can be misleading

Perplexity depends on the definition of a token.

One tokenizer might split a phrase into three tokens. Another might represent the same phrase using five tokens.

Because perplexity averages log-loss per token, changing the token boundaries changes the denominator and the prediction tasks.

A byte-level model can have a much smaller numerical perplexity than a word-level model while being a worse predictor of the same text.

So perplexity comparisons are most meaningful when tokenization, evaluation data, document boundaries, and scoring conventions are controlled.

For comparisons across different tokenizers, we need a shared unit of measurement.

That unit can be a byte.

7. Hands-on experiment: calculate entropy, KL divergence, and cross-entropy on real text

Now we’ll put the mathematics to work.

Instead of synthetic probabilities, we’ll use an actual book and measure how much information its text contains.

Our dataset will be Pride and Prejudice by Jane Austen, available as a UTF-8 plain-text file from Project Gutenberg. The edition’s text file is available from the project’s public archive.

We will build two deliberately simple probability models:

1. A unigram byte model, predicting each byte independently of its context.

2. A bigram byte model, predicting each byte using the immediately preceding byte.

These are not neural networks. They are small statistical language models that make information-theoretic concepts easy to inspect.

Why use bytes instead of characters?

UTF-8 represents text using sequences of bytes. In English, many common characters occupy one byte, while characters such as curly quotation marks and many accented letters use multiple bytes.

A byte can be represented by one of 256 possible values.

This gives us a fixed vocabulary that is independent of subword tokenization. It also establishes a common unit for comparing different models.

We will measure:

• Empirical entropy of the held-out byte distribution.

• Cross-entropy of the held-out bytes under the trained unigram model.

• KL divergence between the held-out byte distribution and the unigram model.

• Cross-entropy of the bigram model on exactly the same held-out bytes.

• Byte-level perplexity.

The complete Python experiment

Save the following as entropy_text.py. It uses only the Python standard library and downloads the source text over HTTPS.

import math
from collections import Counter
from urllib.request import Request, urlopen

URL = "https://www.gutenberg.org/cache/epub/1342/pg1342.txt"

request = Request(
    URL,
    headers={"User-Agent": "EntropyTutorial/1.0"}
)

with urlopen(request, timeout=30) as response:
    source = response.read().decode("utf-8-sig")

start_marker = "*** START OF THE PROJECT GUTENBERG EBOOK"
end_marker = "*** END OF THE PROJECT GUTENBERG EBOOK"

if start_marker not in source or end_marker not in source:
    raise ValueError("Unexpected Project Gutenberg text format")

# Remove Gutenberg's surrounding metadata and license.
body = source.split(start_marker, 1)[1].split("\n", 1)[1]
body = body.split(end_marker, 1)[0]
body = body.replace("\r\n", "\n")

# Treat the text as a sequence of UTF-8 bytes.
bytes_data = body.encode("utf-8")

# Use non-overlapping, chronological sections.
train = bytes_data[:120_000]
test = bytes_data[120_000:160_000]

if len(test) < 40_000:
    raise ValueError("Not enough text for this example")

# Laplace smoothing ensures all 256 byte values
# receive nonzero probabilities.
alpha = 1.0
vocabulary = 256

train_counts = Counter(train)
test_counts = Counter(test)

# Estimate unigram probabilities from training text.
unigram_q = {
    b: (train_counts[b] + alpha)
    / (len(train) + alpha * vocabulary)
    for b in range(vocabulary)
}

# Empirical distribution of held-out bytes.
p_test = {
    b: count / len(test)
    for b, count in test_counts.items()
}

# Entropy of the held-out marginal distribution.
entropy = -sum(
    p * math.log2(p)
    for p in p_test.values()
)

# Cross-entropy under the training unigram model.
cross_entropy = -sum(
    p * math.log2(unigram_q[b])
    for b, p in p_test.items()
)

# KL divergence from held-out P to learned Q.
kl = sum(
    p * math.log2(p / unigram_q[b])
    for b, p in p_test.items()
)

# Verify the mathematical identity.
assert math.isclose(
    cross_entropy,
    entropy + kl,
    abs_tol=1e-10
)

# Train a conditional model:
# P(current byte | previous byte).
pairs = Counter(zip(train[:-1], train[1:]))
previous_counts = Counter(train[:-1])

# Let the first test byte see its actual preceding byte.
continuation = train[-1:] + test

bigram_bpb = sum(
    -math.log2(
        (pairs[(previous, current)] + alpha)
        / (previous_counts[previous] + alpha * vocabulary)
    )
    for previous, current in zip(
        continuation[:-1],
        continuation[1:]
    )
) / len(test)

print(f"Training bytes:            {len(train):,}")
print(f"Held-out bytes:            {len(test):,}")
print(f"Held-out byte entropy:     {entropy:.3f} bits/byte")
print(f"Unigram cross-entropy:     {cross_entropy:.3f} bits/byte")
print(f"Unigram KL divergence:    {kl:.3f} bits/byte")
print(f"Bigram cross-entropy:      {bigram_bpb:.3f} bits/byte")
print(f"Bigram byte perplexity:   {2 ** bigram_bpb:.3f}")
print(f"Uniform baseline:         {math.log2(vocabulary):.1f} bits/byte")

Run it with:

python entropy_text.py

The first two sample sizes should be 120,000 training bytes and 40,000 held-out bytes. The measured information scores are calculated from the downloaded text, so they can change if the source edition or preprocessing changes.

For reference, this is the complete output of our own run (Python 3.12, Project Gutenberg text fetched on 2026-10-15):

Training bytes:            120,000
Held-out bytes:            40,000
Held-out byte entropy:     4.544 bits/byte
Unigram cross-entropy:     4.559 bits/byte
Unigram KL divergence:    0.015 bits/byte
Bigram cross-entropy:      3.670 bits/byte
Bigram byte perplexity:   12.732
Uniform baseline:         8.0 bits/byte

The unigram model sits only 0.015 bits/byte above the held-out entropy, because training and held-out text come from the same book. The bigram model drops to 3.670 bits/byte, below the marginal entropy of 4.544 bits/byte; the next sections explain why that does not break any rule.

Understanding the results

The uniform baseline is exactly 8 bits per byte because there are 256 possible byte values and log₂(256) = 8.

Our unigram model should perform substantially better because actual English text does not use byte values uniformly. Spaces, common letters, and punctuation appear far more frequently than many other byte values.

The empirical entropy of the held-out byte distribution measures the uncertainty of the marginal byte frequencies in that particular sample.

The unigram cross-entropy measures how many bits the trained unigram model assigns, on average, to those same held-out observations.

Their difference is the KL divergence.

Our assertion verifies:

Unigram cross-entropy − held-out byte entropy = unigram KL divergence

The bigram model adds information about the preceding byte.

For example, after seeing a byte representing the letter q, the next byte is much more likely to represent u than a randomly selected letter, at least in this English-language corpus.

A bigram model can exploit that dependence, while an unconditional unigram model cannot.

You should therefore expect the bigram model to achieve a lower held-out cross-entropy, although its actual performance depends on the training sample and smoothing.

An important subtlety about conditional entropy

The bigram model’s cross-entropy can fall below the marginal entropy reported for the test bytes.

That does not contradict the principle that cross-entropy cannot be lower than entropy.

We are now comparing different prediction problems.

The unigram model predicts a byte without context. Its corresponding entropy is H(byte).

The bigram model predicts a byte with the preceding byte available. Its relevant theoretical quantity is H(byte | previous byte).

Conditioning generally reduces uncertainty:

H(byte | previous byte) ≤ H(byte)

A model that knows even more of the preceding text can often reduce uncertainty further.

This is the principle that makes powerful language models useful. They exploit context to make observations less surprising.

What this experiment does not prove

Our evaluation uses a later section of the same book. That is suitable for an introductory demonstration, but it is not evidence of broad generalization.

The text has a consistent author, style, and topic. A model evaluated on an unrelated novel, news article, or technical manual might perform differently.

For a stronger experiment, train on one collection of documents and evaluate on separate documents, ensuring that duplicates and near-duplicates do not cross the split.

The unigram KL calculation is also based on the empirical marginal byte distribution of a finite sample, not the unknown true distribution of natural language.

These limitations do not invalidate the exercise. They clarify exactly which quantities the code measures.

8. Why bits per byte is a language-model compression score

We can now connect the byte-level experiment to large language models.

A language model assigns a probability to each observed next token.

Those probabilities can be converted into ideal code lengths using negative base-2 logarithms.

If a token receives probability 1/2, its theoretical coding cost is 1 bit. If it receives probability 1/256, its cost is 8 bits.

Across a complete sequence, the total ideal coding cost is:

Total bits = −Σ log₂ q(actual next token | context)

This is the sum of the same log-loss terms used during training, but expressed in bits rather than nats.

An entropy coder, such as an appropriate arithmetic or range coder, can use these probability predictions to approach this coding cost over long sequences.

Now divide the total by the number of UTF-8 bytes in the original text.

Bits per byte (BPB) = total predicted code length in bits / original text bytes

If we already have a model’s mean token cross-entropy in nats, the conversion is:

BPB = (mean loss in nats × scored token count) / (ln(2) × original byte count)

This normalization matters because different language models often use different tokenizers.

A model might represent a piece of text with 100 tokens, while another represents it with 140. Their mean per-token cross-entropies are not directly comparable.

But if both score the exact same UTF-8 text with equivalent probability accounting, the total predicted code length can be normalized by the same number of original bytes.

That gives a more meaningful comparison of compression efficiency.

What does a score of 1 bit per byte mean?

Raw bytes contain 8 bits each.

A model achieving 1 bit per byte assigns an ideal average code length one-eighth of the raw byte representation.

In other words, it suggests an ideal compression ratio of approximately 8:1 on that evaluated text.

Some example rates:

These are idealized coding ratios, not actual archive file sizes.

Practical compression has overhead. In particular, the decoder needs access to the probability model, and storing a large language model’s parameters may cost vastly more than the text being compressed.

The ratio also assumes that the exact scored probability distribution can be reproduced during decoding and that all relevant symbols and context are accounted for.

Therefore, BPB measures predictive compression efficiency, not necessarily the size of a usable compressed file.

This distinction is especially important for gigantic language models.

9. The cherry on the cake: modern LLMs are remarkably effective text compressors

Here is the surprising result: a language model trained to predict tokens can become an extraordinarily efficient statistical model of human writing, even though its principal purpose is not to produce compressed archives.

In July 2026, Trelis Research published BPBench, a compression-oriented evaluation of large language models on fresh human-authored prose.

The benchmark uses exact model token log-probabilities and normalizes the resulting negative log-likelihood by the number of original UTF-8 bytes. Its evaluation includes 457 documents across three slices: UK parliamentary speech, US congressional speech, and a private prose collection.

What do strong 2026 models achieve?

The following are selected results reported by BPBench, using the macro-average of the three prose slices.

Source: Trelis Research, BPBench, July 2026. Lower is better. Values are benchmark-specific estimates, not universal model compression rates. Selected language models: compression of human proseMacro-average bits per byte from BPBench, July 2026. Lower is better.

A score of 0.615 bits per byte corresponds to an ideal raw-byte compression ratio of approximately 13:1.

That’s an extraordinary level of predictability, especially considering that the evaluation uses held-out natural prose rather than a specially constructed repetitive dataset.

But the precise ordering deserves caution.

The leading models have overlapping confidence intervals. BPBench’s authors specifically warn against treating small score differences as definitive rankings. Results also depend on the document selection, checkpoint, serving precision, and post-training choices.

NVIDIA’s Nemotron 3 Ultra, for example, is a large mixture-of-experts model whose release was documented in June 2026. The model’s capabilities and training configuration are independent of the compression benchmark, which measures its predictive behavior on a particular evaluation corpus.

Why compression is such an interesting measure of model quality

A language model that assigns high probability to the correct continuation must capture meaningful regularities in text.

Some of those regularities are relatively simple:

• Common characters and word fragments.

• Spelling and punctuation conventions.

• Frequent grammatical constructions.

Others require progressively richer context:

• Which entities a paragraph discusses.

• What a speaker has already said.

• How a technical explanation is structured.

• Which words make sense given the subject and surrounding argument.

A powerful predictor can exploit all of these patterns to reduce uncertainty.

However, low BPB should not be confused with a complete measurement of intelligence or assistant usefulness.

Two models with similar cross-entropy might differ substantially in reasoning, tool use, instruction following, factual reliability, or safety behavior.

Post-training can also alter token probabilities in ways that affect compression even when the model becomes more useful as an assistant.

Compression measures the quality of a probability model on a distribution of sequences. It does not measure every capability we expect from a capable language model.

That distinction makes BPB scientifically valuable without turning it into another universal leaderboard.

10. What these measurements teach us about training

Information theory offers several practical lessons for machine-learning engineering.

Lower training loss does not guarantee better generalization

A sufficiently flexible model can memorize training examples.

That can make its training cross-entropy extremely small even when its probabilities on new text are poor.

Evaluate on held-out data, preferably documents separated at the source level, not merely randomly selected tokens from the same documents.

Logarithm bases determine the unit, not the ranking

Cross-entropy calculated with natural logarithms is measured in nats.

Cross-entropy calculated with base-2 logarithms is measured in bits.

The conversion is:

Bits = nats / ln(2)

Changing the logarithm base multiplies every score by the same positive constant, so it does not change which model has the lower loss on the same scoring problem.

Smoothing prevents infinite penalties in small statistical models

Our byte-level experiment used Laplace smoothing.

Without smoothing, an unseen byte transition would receive probability zero from a simple frequency estimator.

The negative logarithm of that probability is infinite.

Smoothing assigns nonzero probability to unseen events, making held-out cross-entropy finite.

Modern neural language models typically produce softmax probabilities across their vocabulary, but numerical precision and implementation details still matter when scoring extremely unlikely outcomes.

Be careful with probability calibration

A model can make accurate top-1 predictions while assigning poor probabilities.

Cross-entropy detects this because it measures the entire probabilistic commitment to the observed outcome.

That makes it useful for applications where confidence matters, including recommendation ranking, uncertainty estimation, and probabilistic forecasting.

However, good aggregate cross-entropy does not automatically guarantee good calibration for every subgroup or context. Calibration should also be evaluated directly where it matters.

Compare models on equivalent evaluation units

For token-based models, report the tokenizer and the evaluation convention alongside perplexity.

For cross-tokenizer comparisons, consider bits per byte, ensuring that models score exactly the same underlying text, that tokenization is reversible for that text, and that initial-context and special-token handling are consistent.

Normalization by bytes does not repair an evaluation whose probability accounting is incomplete.

11. Exercises to deepen your understanding

The best way to internalize these concepts is to experiment with models whose probability distributions you can inspect.

Here are four exercises that extend the Python experiment.

Exercise 1: Change the smoothing strength. Try setting the smoothing parameter to 0.1, 1.0, and 10.0. Observe how unigram and bigram cross-entropy change. Explain why heavy smoothing can protect against unseen events but also flatten genuinely informative probability differences.

Exercise 2: Increase the context length. Build a trigram byte model that conditions on the preceding two bytes. Compare its held-out BPB with the unigram and bigram models. Does the extra context always help with limited training data? Why might a larger context model suffer from sparse observations?

Exercise 3: Evaluate different writing styles. Train your byte model on one Project Gutenberg novel and test on another public-domain book. Compare this result with the within-book evaluation. Look for changes caused by vocabulary, punctuation, formatting, and authorial style.

Exercise 4: Connect BPB to tokenizer-based loss. Take a locally available language model that exposes next-token log-probabilities. Score a fixed passage, sum the log-probabilities of every observed token under teacher forcing, and divide by the UTF-8 byte count. Compare the result with its token perplexity. Document exactly how you handle the first token and context boundaries.

These exercises reveal a central tension in language modeling: richer context offers more predictive information, but learning reliable probability estimates from that context requires sufficient data and appropriate model capacity.

12. Final mental model: loss is the cost of describing data

We can now connect the entire picture.

Entropy measures the uncertainty inherent in a probability distribution.

Cross-entropy measures the expected information cost when predictions come from a possibly incorrect model.

KL divergence measures the excess cost created by that mismatch.

Negative log-likelihood turns those quantities into a practical training objective.

Perplexity exponentiates average cross-entropy into a more intuitive effective uncertainty scale.

Bits per byte normalizes the sequence’s predicted code length by its original byte count, enabling more meaningful comparison across different tokenizers under controlled evaluation.

The relationships worth remembering are:

The most important insight is that a model learns to predict by learning a more efficient probabilistic description of its data.

Better next-token predictions reduce log-loss. Lower log-loss means shorter ideal codes. And shorter ideal codes mean the model has discovered statistical structure that makes the data more predictable.

That is why a loss function from information theory sits at the center of modern language-model training.

Your next step: run the real-text experiment, then extend it from unigram and bigram predictions to a trigram model. Record the held-out cross-entropy, KL divergence where applicable, and bits per byte. The moment you see additional context reduce the cost of predicting unseen text, the connection between entropy, learning, and compression becomes concrete.