Probability for ML: Bayes’ rule, likelihood, and why models output logits

Machine learning models often appear mysterious at the point where they produce an answer. A classifier receives an input, performs millions or billions of calculations, and returns something like “spam: 97%” or “cat: 83%.” Behind that output is not magic. It is a chain of probability concepts that starts with a simple question:

Given the evidence we have observed, what should we believe?

That question connects classical probability, Bayes’ rule, maximum likelihood estimation, neural network logits, the softmax function, and cross-entropy loss. Understanding this chain gives you a practical mental model for why modern classification systems are trained the way they are.

This course-style walkthrough builds the ideas step by step:

• How Bayes’ rule updates beliefs using evidence

• Why likelihood is central to machine learning training

• How models transform raw numbers into probabilities

• Why neural networks output logits instead of probabilities directly

• How softmax converts logits into class predictions

• Why cross-entropy is the natural loss function for classification

The goal is not just to memorize formulas. The goal is to understand what the model is trying to optimize.

The starting point: probability is about uncertainty

A machine learning model rarely has perfect certainty. Even a powerful image classifier may encounter a blurry photo, an unusual object, or an example it has never seen before.

Probability gives models a language for uncertainty.

Suppose an email filter receives a message. It wants to estimate:

• How likely is this email to be spam?

• How likely is this email to be legitimate?

Before seeing the email content, the system might already know something from historical data. For example, if 30% of incoming emails are spam, the initial probability of spam is 30%.

This initial belief is called a prior probability.

After reading the email, the model updates that belief using evidence:

• The sender address

• Words in the subject line

• Links inside the message

• Writing patterns

• Previous behavior associated with similar messages

The updated belief is the posterior probability.

This is the core idea behind Bayes’ rule.

Bayes’ rule: updating beliefs with evidence

Bayes’ rule describes how to update a probability after observing new information:

P(A | B) = P(B | A) × P(A) / P(B)

In words:

The probability of A given B equals the probability of seeing B if A is true, multiplied by the original probability of A, divided by the overall probability of seeing B.

For machine learning, the terms usually map like this:

• A is a hypothesis, such as “this email is spam”

• B is evidence, such as “this email contains suspicious links”

• P(A) is the prior probability

• P(B | A) is the likelihood

• P(A | B) is the updated belief after seeing evidence

The likelihood is especially important. It answers:

“If this hypothesis were true, how expected would this evidence be?”

Machine learning models spend much of their time estimating relationships like this.

A worked example: why base rates matter

Here is a surprising case where intuition often fails.

Imagine a fraud detector for online payments.

The system is extremely accurate:

• It correctly detects fraud 99% of the time when a fraudulent transaction occurs.

• It correctly ignores legitimate transactions 99% of the time.

That sounds impressive. Now imagine fraud is rare:

• Only 1 out of every 1,000 transactions is actually fraudulent.

A customer receives a fraud alert. What is the probability that the transaction is truly fraud?

Many people guess around 99%. The real answer is much lower.

Consider 100,000 transactions:

• 100 transactions are fraudulent.

• 99 of those are detected.

• 99,900 transactions are legitimate.

• 1% of legitimate transactions are incorrectly flagged.

• That creates 999 false alarms.

The alert group contains:

• 99 true fraud cases

• 999 false alarms

So the probability that an alert is actually fraud is:

99 / (99 + 999) ≈ 9%

This is the base-rate fallacy. The accuracy of the detector is not enough. The underlying frequency of the event matters.

This is why probability models need careful calibration. A model can have excellent accuracy metrics and still produce misleading predictions if the probabilities do not reflect reality.

Likelihood: the engine behind machine learning training

Bayes’ rule uses likelihood, but machine learning often approaches the problem from the other direction.

Instead of asking:

“Given this model, how likely is the data?”

training asks:

“Which model parameters make the observed data most likely?”

This is maximum likelihood estimation.

Suppose we have a simple model predicting whether an online store customer will purchase an item.

The dataset might contain:

visitor_age, visits_last_month, purchased
24, 3, 0
31, 8, 1
45, 2, 0
29, 12, 1

The model has parameters that control its predictions. Training adjusts those parameters so the predictions become more consistent with the observed examples.

The model is effectively searching for parameters that maximize:

Probability of the observed training data given the model

A neural network may have millions or billions of parameters, but the principle is the same.

From model scores to probabilities

A classifier usually does not begin by producing probabilities.

Instead, it produces raw scores.

For a simple three-class image classifier, the network might output:

class_0: 4.2
class_1: 1.7
class_2: -0.5

These numbers are called logits.

A logit is an unnormalized score. It does not have a direct probability interpretation.

A higher logit means the model has stronger evidence for that class relative to the others.

The model might know:

• Class 0 looks much more likely than class 1

• Class 1 looks somewhat possible

• Class 2 looks unlikely

But the raw values do not add up to 1 and cannot be interpreted as percentages.

The conversion from logits to probabilities happens later.

Why models output logits instead of probabilities

It may seem simpler to make the final neural network layer output probabilities directly. However, logits are more useful during training.

There are several reasons:

1. Better numerical stability

Probabilities involve operations like exponentials and divisions. With very large or very small values, calculations can become unstable.

Working with logits allows software libraries to combine operations in numerically safer ways.

2. More flexible optimization

The model can freely increase or decrease confidence through the scale of logits.

A class with a logit of 10 is not just “more likely” than a class with a logit of 2. The difference represents a much stronger preference after transformation.

3. Loss functions expect logits

Modern deep learning frameworks usually provide combined loss functions that accept logits directly and internally apply the required transformation.

For example, in Python with a common deep learning workflow, the model output might look like this:

import torch
import torch.nn as nn

logits = torch.tensor([[2.5, 0.5, -1.0]])

target = torch.tensor([0])

loss_function = nn.CrossEntropyLoss()
loss = loss_function(logits, target)

print(loss.item())

The model does not need to manually calculate probabilities before calculating the loss.

Softmax: turning logits into probabilities

For multi-class classification, the softmax function converts logits into probabilities.

The idea:

1. Exponentiate each logit.

2. Add all exponentiated values together.

3. Divide each exponentiated value by the total.

The result:

• Every value is between 0 and 1.

• All values add up to 1.

The relationship can be written as:

Softmax(zᵢ) = exp(zᵢ) / Σ exp(zⱼ)

where z represents the logits.

For example:

import torch

logits = torch.tensor([4.0, 2.0, 1.0])

probabilities = torch.softmax(logits, dim=0)

print(probabilities)
print(probabilities.sum())

A possible output:

tensor([0.8438, 0.1142, 0.0420])
tensor(1.)

The largest logit becomes the largest probability, but the smaller classes still receive some probability mass.

That matters because uncertainty is useful information.

A model that outputs:

dog: 0.51
wolf: 0.46
car: 0.03

is telling us something very different from:

dog: 0.99
wolf: 0.005
car: 0.005

Even though both predict “dog.”

Cross-entropy: measuring prediction quality

Once a model produces probabilities, we need a way to measure how good those predictions are.

Cross-entropy does this by rewarding confident correct predictions and strongly penalizing confident wrong predictions.

Imagine a model classifying an image:

Prediction A:

cat: 0.95
dog: 0.05

Prediction B:

cat: 0.55
dog: 0.45

If the image is actually a cat, both are correct, but Prediction A is more useful because it expresses stronger evidence.

Now consider:

cat: 0.01
dog: 0.99

A confident wrong answer should receive a large penalty.

During training, gradient descent adjusts the model parameters to reduce cross-entropy. The network gradually learns which patterns increase the probability of the correct class.

This creates a direct connection:

• The model produces logits.

• Softmax interprets those logits as probabilities.

• Cross-entropy measures whether those probabilities match reality.

• Optimization updates the model to improve future predictions.

A complete mini example: training a classifier

Here is a small synthetic example using generated data. It avoids external datasets and creates a simple classification problem:

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=500,
    n_features=4,
    n_classes=3,
    n_informative=3,
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

model = LogisticRegression(max_iter=1000)

model.fit(X_train, y_train)

probabilities = model.predict_proba(X_test[:5])

print(probabilities)
print(probabilities.sum(axis=1))

The output probabilities for each example should sum to approximately 1.

Although logistic regression is much simpler than a deep neural network, the probability pipeline is similar:

• Learn parameters from data.

• Produce scores.

• Convert scores into probabilities.

• Evaluate predictions.

The bigger picture: probability connects every ML layer

Modern artificial intelligence systems often look complicated because they combine many components:

• Neural networks

• Attention mechanisms

• Large-scale optimization

• Distributed training

• Massive datasets

But underneath, many systems still rely on the same foundational ideas:

• Represent uncertainty.

• Estimate likelihood.

• Compare predictions with observations.

• Update parameters to improve future predictions.

Understanding probability turns machine learning from a collection of techniques into a coherent system.

When you understand why a model outputs logits, you are no longer treating the final layer as a mysterious black box. You can trace the entire path:

Evidence → model parameters → logits → probabilities → decision

That path is the mathematical backbone of classification.

Key takeaways

Remember these principles:

• Bayes’ rule explains how evidence updates beliefs.

• The prior probability matters, especially for rare events.

• Likelihood measures how compatible evidence is with a hypothesis.

• Machine learning training often follows maximum likelihood principles.

• Logits are raw model scores, not probabilities.

• Softmax converts logits into a probability distribution.

• Cross-entropy trains classifiers by rewarding accurate confidence.

• Good probability estimates require more than high accuracy.

Probability is not an optional extra layer added after machine learning. It is the language that explains what machine learning models are trying to learn.

Start with Bayes’ rule, experiment with logits and softmax, and build your intuition by implementing small models yourself. The fastest way to understand modern ML is to connect the equations, the code, and the predictions into one continuous story.