Metrics that match the problem: ROC-AUC vs F1 vs MAE vs RMSE, and when accuracy lies

A machine learning model achieves 99% accuracy. Its dashboard looks impressive. Stakeholders are satisfied, and the team is preparing to deploy it.

There is just one problem: the model misses every fraudulent transaction.

Meanwhile, another model achieves only 97.8% accuracy but successfully identifies 80% of fraudulent transactions. Which model is better?

The answer depends on what you’re trying to accomplish. And that is exactly why choosing the right evaluation metric matters as much as choosing the right algorithm.

Machine learning metrics are not interchangeable. Accuracy, ROC-AUC, F1, MAE, and RMSE answer fundamentally different questions:

• Accuracy: How often are the model’s classifications correct?

• ROC-AUC: How well does a classifier rank positive examples above negative examples?

• F1: How effectively does a classifier balance precision and recall at a chosen decision threshold?

• MAE: How large is the average absolute prediction error?

• RMSE: How large is the typical prediction error when larger mistakes receive disproportionately more weight?

A metric can be mathematically correct while telling you almost nothing useful about whether a system will succeed in production.

In this tutorial, you’ll learn how these metrics work, implement them using modern scikit-learn, reproduce several surprising results, and build a practical framework for choosing evaluation metrics based on the problem you’re solving.

By the end, you’ll understand not just how to calculate a metric, but when to trust it, when to question it, and what to measure alongside it.

1. First, identify the type of machine learning problem

Before comparing metrics, determine what your model predicts.

Most supervised learning problems fall into two categories.

Classification: predicting categories

A classification model assigns an observation to a category.

Examples include:

• Detecting fraudulent versus legitimate transactions.

• Classifying emails as spam or not spam.

• Predicting whether a customer will cancel a subscription.

• Identifying defective versus acceptable manufactured products.

• Categorizing customer support tickets.

A binary classifier predicts one of two classes, often represented as 0 and 1.

A multiclass classifier chooses among three or more categories, while a multilabel classifier can assign several labels to the same example.

Common classification metrics include accuracy, precision, recall, F1, and ROC-AUC.

Regression: predicting numbers

A regression model predicts a continuous numerical value.

Examples include:

• Forecasting daily electricity consumption.

• Estimating apartment rental prices.

• Predicting delivery time in minutes.

• Forecasting retail revenue.

• Estimating server response latency.

Common regression metrics include MAE, MSE, RMSE, and R².

The distinction is important because classification and regression metrics generally cannot be substituted for each other.

A classification model might be wrong because it predicts the incorrect category. A regression model might be wrong by 2 units or 200 units.

Those errors need different ways of being measured.

2. Set up a reproducible Python environment

We’ll use NumPy and scikit-learn throughout the tutorial. Every dataset in the runnable examples is either constructed directly or generated synthetically, so no downloads, credentials, or external data files are required.

Create a project directory and a virtual environment:

mkdir model-metrics-lab
cd model-metrics-lab
python -m venv .venv

Activate it on macOS or Linux:

source .venv/bin/activate

Install the dependencies:

python -m pip install --upgrade pip
python -m pip install --upgrade numpy scikit-learn

Check your installed versions:

python -m pip show numpy scikit-learn

The examples use APIs supported by current scikit-learn releases, including root_mean_squared_error and TunedThresholdClassifierCV. The latter is useful for separating model training from the decision about which probability threshold should trigger an action.

The official scikit-learn metrics reference documents these functions and their supported arguments.

Each example below is self-contained and can be saved as its own Python file.

3. When 99% accuracy is completely useless

Accuracy is probably the most familiar machine learning evaluation metric.

Its formula is straightforward:

Accuracy = Number of correct predictions / Total number of predictions

For binary classification, we can write:

Accuracy = (TP + TN) / (TP + TN + FP + FN)

The four quantities are:

A model with 95% accuracy correctly classifies 95 out of every 100 examples, assuming all examples have equal weight.

That sounds useful, but accuracy has a critical weakness: it treats every correct prediction as equally valuable, regardless of which class it belongs to.

The class imbalance problem

Imagine an e-commerce platform processes 10,000 orders.

Of those orders:

• 9,900 are legitimate.

• 100 are fraudulent.

• Fraud therefore represents exactly 1% of the dataset.

Now imagine a classifier that always predicts “legitimate.”

It never analyzes suspicious behavior. It never identifies fraud. It simply predicts the majority class for every transaction.

Its results are:

The accuracy is impressive, but the fraud-detection performance is useless.

This is called the accuracy paradox: a model can achieve a high overall accuracy score while performing poorly on the class that matters most.

Runnable example: the useless 99% classifier

Let’s reproduce this with scikit-learn and compare it with another hypothetical classifier that catches most fraud but produces some false alarms.

import numpy as np

from sklearn.metrics import (
    accuracy_score,
    precision_score,
    recall_score,
    f1_score,
    confusion_matrix,
)

# 10,000 synthetic orders: 1% fraudulent.
y_true = np.array([0] * 9900 + [1] * 100)

# Model A: always predicts legitimate.
model_a = np.zeros_like(y_true)

# Model B: flags 80 fraudulent orders,
# but also flags 200 legitimate orders.
model_b = np.zeros_like(y_true)
model_b[:200] = 1
model_b[9900:9980] = 1

def evaluate(name, predictions):
    print(f"\n{name}")
    print(f"Accuracy:  {accuracy_score(y_true, predictions):.2%}")
    print(
        f"Precision: "
        f"{precision_score(y_true, predictions, zero_division=0):.2%}"
    )
    print(
        f"Recall:    "
        f"{recall_score(y_true, predictions, zero_division=0):.2%}"
    )
    print(
        f"F1:        "
        f"{f1_score(y_true, predictions, zero_division=0):.3f}"
    )
    print("Confusion matrix:")
    print(confusion_matrix(y_true, predictions))

evaluate("Model A: Always legitimate", model_a)
evaluate("Model B: Fraud review model", model_b)

The results are:

Model B has lower accuracy, yet it detects 80 fraudulent orders that Model A completely misses.

Whether Model B is commercially worthwhile depends on the cost of investigating its 200 false alarms versus the value of detecting 80 fraudulent orders.

But one thing is clear: maximizing accuracy would select Model A, even though it provides no fraud-detection capability.

The confusion matrix makes this distinction immediately visible.

Model A’s matrix is:

[[9900    0]
 [ 100    0]]

Model B’s matrix is:

[[9700  200]
 [  20   80]]

In scikit-learn’s standard binary confusion-matrix layout, the first row represents actual negatives, the second represents actual positives, and the columns represent predicted negatives and positives.

Reading the matrix is often more informative than reading a single metric.

When accuracy is still useful

Accuracy is not inherently bad.

It works well when:

• Class frequencies are reasonably balanced.

• False positives and false negatives have similar costs.

• Every classification has roughly equal importance.

• The primary goal is maximizing the proportion of correct labels.

For example, consider a system classifying manufactured items into four equally common product categories. If every misclassification creates approximately the same operational cost, accuracy can be a reasonable primary metric.

The lesson is not to abandon accuracy. It is to stop using accuracy without understanding class distribution and error costs.

4. Precision, recall, and F1: measuring what accuracy misses

To evaluate imbalanced classification problems, we often need to examine errors involving the positive class more carefully.

Three metrics are particularly useful.

Precision: how trustworthy are positive predictions?

Precision measures the fraction of predicted positives that are actually positive.

Precision = TP / (TP + FP)

For fraud detection, precision answers:

Of all the transactions we flagged as fraudulent, how many really were fraudulent?

In Model B:

• True positives = 80

• False positives = 200

Therefore:

Precision = 80 / (80 + 200) = 0.2857

Only about 28.6% of the flagged transactions are fraudulent.

That means the investigation team receives roughly 2.5 false alarms for every genuine fraud case it detects.

High precision is particularly important when acting on a false alarm is expensive.

Examples include:

• Automatically blocking customer accounts.

• Suspending legitimate marketplace sellers.

• Removing content from an online platform.

• Triggering costly manual investigations.

Recall: how many actual positives did we find?

Recall measures the fraction of actual positives that the model successfully identifies.

Recall = TP / (TP + FN)

For fraud detection, recall answers:

Of all the fraudulent transactions, how many did we catch?

In Model B:

• True positives = 80

• False negatives = 20

Therefore:

Recall = 80 / (80 + 20) = 0.80

The model detects 80% of fraud cases.

A high-recall classifier catches a large fraction of the events we care about, even if it generates additional false alarms.

This can be appropriate when missing a positive case is especially costly, such as missing a major security incident or allowing a fraudulent transfer to complete.

F1: balancing precision and recall

F1 combines precision and recall using their harmonic mean.

F1 = 2 × (Precision × Recall) / (Precision + Recall)

An equivalent formula is:

F1 = 2TP / (2TP + FP + FN)

F1 ranges from 0 to 1, with larger values indicating a better balance between precision and recall.

For Model B:

F1 = 2 × (0.2857 × 0.80) / (0.2857 + 0.80) ≈ 0.421

Why use the harmonic mean rather than the arithmetic mean?

Because F1 strongly penalizes an extreme imbalance between precision and recall.

For example, a classifier with precision of 1.0 and recall of 0.1 has an F1 score of approximately 0.182, not 0.55.

A classifier cannot achieve a high F1 score simply by making one component excellent while nearly ignoring the other.

The precision-recall trade-off

Many binary classifiers do not inherently predict a class. Instead, they produce a probability or decision score.

We then apply a threshold.

For a model producing estimated probabilities of fraud:

• A threshold of 0.9 flags only very high-confidence cases.

• A threshold of 0.5 flags cases estimated to have at least 50% fraud probability.

• A threshold of 0.1 flags many more suspicious cases.

Lowering the threshold typically increases recall, but may also increase false positives and reduce precision.

Raising the threshold typically increases precision, but may cause the model to miss more genuine positives. These are common tendencies, not guarantees for every finite dataset.

This means F1 is a property of the model together with its chosen decision threshold, not just its underlying ranking quality.

There is also an important limitation: F1 ignores true negatives entirely.

For rare-event detection, that is often useful. But if the business cost of different errors is highly asymmetric, even F1 may not be the correct objective.

For example, when one missed fraudulent transfer costs significantly more than several hundred investigations, recall or a monetary cost function may be more appropriate than F1.

5. ROC-AUC: evaluating ranking rather than one decision threshold

Now suppose you have two classifiers that output probability-like scores.

You want to know which model generally assigns higher scores to positive examples than to negative ones, regardless of a particular classification cutoff.

That is where ROC-AUC becomes useful.

Understanding the ROC curve

ROC stands for Receiver Operating Characteristic.

The ROC curve plots:

• True positive rate (TPR): TP / (TP + FN), which is recall.

• False positive rate (FPR): FP / (FP + TN).

The curve shows how these quantities change as the classification threshold moves.

The area under this curve is ROC-AUC.

For binary classification, an intuitive interpretation is:

ROC-AUC is the probability that a randomly selected positive example receives a higher model score than a randomly selected negative example, with ties receiving half credit.

Important reference points:

A score of 0.8 can be promising, but it is not a universal standard for an acceptable model.

The business problem, class prevalence, and allowed false-positive rate all matter.

Runnable example: ROC-AUC and F1 disagree

Here is an intentionally simple experiment with 100 synthetic transactions, 10 of which are fraudulent.

We’ll compare two scoring models.

Model A produces excellent rankings but assigns every observation a score below 0.5.

Model B ranks the observations less perfectly but identifies all positives when we apply the default 0.5 threshold.

import numpy as np

from sklearn.metrics import (
    roc_auc_score,
    average_precision_score,
    accuracy_score,
    f1_score,
)

# 90 legitimate transactions, 10 fraudulent.
y_true = np.array([0] * 90 + [1] * 10)

# Model A: perfect ranking.
# All positives score higher than all negatives,
# but every score is below 0.5.
scores_a = np.array([0.10] * 90 + [0.40] * 10)

# Model B: 10 negatives score above the positives.
# It still identifies all positives at threshold 0.5.
scores_b = np.array(
    [0.10] * 80 +
    [0.80] * 10 +
    [0.70] * 10
)

def evaluate_scores(name, scores, threshold=0.5):
    predictions = (scores >= threshold).astype(int)

    print(f"\n{name}")
    print(f"ROC-AUC: {roc_auc_score(y_true, scores):.3f}")
    print(
        f"AP:      "
        f"{average_precision_score(y_true, scores):.3f}"
    )
    print(
        f"Accuracy: "
        f"{accuracy_score(y_true, predictions):.3f}"
    )
    print(f"F1:      {f1_score(y_true, predictions):.3f}")

evaluate_scores("Model A", scores_a)
evaluate_scores("Model B", scores_b)

adjusted_predictions = (scores_a >= 0.30).astype(int)

print(
    "\nModel A F1 at threshold 0.30:",
    f1_score(y_true, adjusted_predictions),
)

The results are:

Model A has perfect ROC-AUC but zero F1 at the default threshold.

Why?

Because its positive scores are 0.40 and its negative scores are 0.10. The ranking is perfect, but none of the positive scores reaches the 0.5 classification threshold.

Model B identifies all 10 positives at that threshold. However, it also flags 10 negatives.

Its precision is 0.5, recall is 1.0, and F1 is approximately 0.667.

Now look at what happens when we lower Model A’s threshold from 0.5 to 0.3.

Its F1 becomes 1.0.

We did not retrain the classifier or change its ranking. We only changed how its scores are converted into decisions.

This demonstrates two important principles:

1. A model with higher ROC-AUC can have lower F1 at a particular threshold.

2. A poor F1 score does not necessarily mean the model learned a poor ranking.

The constructed scores in this experiment are illustrative, not probabilities calibrated against observed fraud rates. In a real deployment, a threshold of 0.3 must be justified through validation and operating costs rather than chosen merely because it produces a convenient result.

When ROC-AUC is useful

ROC-AUC is especially useful when:

• Comparing models before selecting an operating threshold.

• Measuring ranking quality over a wide range of thresholds.

• Evaluating systems where risk scores are consumed by another decision process.

• Comparing classifiers whose predicted probabilities have different scales.

However, ROC-AUC has limitations.

A classifier can perform well across the full ROC curve while performing inadequately in the very-low-false-positive region where a production system must operate.

It also does not directly answer:

• How many alerts will human analysts investigate?

• What fraction of flagged cases will be genuine?

• How many fraud cases can we catch within our review budget?

• Are predicted probabilities well calibrated?

These questions require additional metrics.

The current scikit-learn documentation for ROC-AUC confirms that binary ROC-AUC accepts probability estimates or non-thresholded decision scores. For standard binary probability outputs, you normally pass the probability of the positive class, not hard predicted labels.

6. Why precision-recall metrics matter for rare events

In a heavily imbalanced dataset, the number of negatives may vastly exceed the number of positives.

Consider another example with 100,000 transactions:

• 99,000 are legitimate.

• 1,000 are fraudulent.

• Fraud prevalence is 1%.

Suppose a model catches 900 fraudulent transactions while incorrectly flagging 4,950 legitimate transactions.

Its recall is:

Recall = 900 / 1,000 = 90%

Its false positive rate is:

FPR = 4,950 / 99,000 = 5%

A 5% false positive rate might sound relatively small. But the model generates 5,850 alerts, of which only 900 are genuine fraud cases.

Its precision is:

Precision = 900 / 5,850 ≈ 15.4%

In other words, approximately 84.6% of its alerts are false alarms.

The problem is not that the false positive rate formula is incorrect. The problem is that a seemingly modest false positive rate can translate into a huge number of false alarms when the negative class dominates.

Average precision and the precision-recall curve

A precision-recall curve plots precision against recall as the decision threshold changes.

Rather than asking how recall changes with false positive rate, it asks how much precision is retained while retrieving more positives.

In scikit-learn, average_precision_score summarizes this relationship using a recall-weighted average of precision values across thresholds.

Average precision (AP) is not identical to the trapezoidal area under a precision-recall curve. This distinction matters when comparing results from different libraries or papers.

In the earlier runnable experiment:

• Model A achieves AP = 1.0.

• Model B achieves AP = 0.5.

That makes sense: Model A always ranks positives above negatives, while Model B puts 10 negatives above all 10 positives.

For a random ranking, the positive-class prevalence provides the usual baseline expectation for average precision. In a dataset with 1% positives, an AP close to 0.01 is therefore not impressive, while an AP substantially above it indicates useful ranking information.

Which should you choose: ROC-AUC or average precision?

Use ROC-AUC when broad ranking discrimination is the main question.

Prefer average precision or a precision-recall curve when:

• Positives are rare.

• Your workflow acts on positive predictions.

• The number of false alarms matters directly.

• You need to evaluate how effectively a model finds a relatively small positive class.

For production fraud detection, monitoring both can be valuable, but neither should replace metrics measured at the actual operating threshold.

7. Choosing the right threshold with scikit-learn

Our earlier experiment demonstrated that the same classifier can achieve F1 = 0.0 or F1 = 1.0 depending on its threshold.

In a real machine learning project, choosing that threshold is part of model development.

The traditional default of 0.5 is not necessarily optimal, especially when the positive class is rare.

A practical threshold-tuning experiment

Let’s train a logistic regression classifier on synthetic e-commerce transaction features.

We’ll create a dataset containing 12,000 observations with a 1% positive class.

Instead of manually adjusting the threshold until the test-set score looks good, we’ll use TunedThresholdClassifierCV, which uses cross-validation to optimize a classification threshold.

This is supported in current scikit-learn releases and avoids the need to implement the threshold search from scratch.

from sklearn.datasets import make_classification
from sklearn.model_selection import (
    train_test_split,
    TunedThresholdClassifierCV,
)
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import (
    f1_score,
    roc_auc_score,
    average_precision_score,
)

# Generate synthetic transaction features.
X, y = make_classification(
    n_samples=12000,
    n_features=10,
    n_informative=6,
    n_redundant=1,
    n_clusters_per_class=1,
    weights=[0.99, 0.01],
    class_sep=1.0,
    flip_y=0,
    random_state=7,
)

# Keep the final test set separate.
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    stratify=y,
    random_state=42,
)

def build_model():
    return make_pipeline(
        StandardScaler(),
        LogisticRegression(
            max_iter=1000,
            random_state=42,
        ),
    )

# Baseline: standard classification threshold.
baseline = build_model()
baseline.fit(X_train, y_train)

baseline_predictions = baseline.predict(X_test)
baseline_scores = baseline.predict_proba(X_test)[:, 1]

# Tune the threshold using training-data CV.
tuned = TunedThresholdClassifierCV(
    estimator=build_model(),
    scoring="f1",
    cv=3,
    thresholds=100,
)

tuned.fit(X_train, y_train)

tuned_predictions = tuned.predict(X_test)

print(f"Positive rate: {y_test.mean():.2%}")

print(
    f"ROC-AUC: "
    f"{roc_auc_score(y_test, baseline_scores):.3f}"
)

print(
    f"Average precision: "
    f"{average_precision_score(y_test, baseline_scores):.3f}"
)

print(
    f"Baseline F1: "
    f"{f1_score(y_test, baseline_predictions):.3f}"
)

print(
    f"Tuned F1: "
    f"{f1_score(y_test, tuned_predictions):.3f}"
)

print(f"Selected threshold: {tuned.best_threshold_:.3f}")

A verified run of this seeded example produces approximately:

Numerical results may vary slightly across compatible scikit-learn versions.

The significant observation is that the classifier’s F1 improves without changing its underlying learning algorithm.

The threshold search changes the final decision rule.

Why the train-test separation matters

There is a tempting but incorrect approach to threshold tuning:

1. Evaluate the classifier on the test set.

2. Try many thresholds.

3. Select the threshold producing the highest test F1.

4. Report that same F1 as the final result.

This leaks information from the test set into the model-selection process.

Even though the classifier’s learned coefficients might remain unchanged, the decision rule has been optimized on data that was supposed to provide an independent evaluation.

That makes the reported score optimistic.

The correct approach is to use training data and internal validation to choose the threshold, then evaluate the complete finalized classifier on an untouched test set.

For real fraud systems, choosing the threshold to maximize F1 might still be inappropriate.

Instead, you might optimize an objective such as:

• Maximize recall while keeping precision above 80%.

• Capture as much fraud as possible with no more than 500 daily investigations.

• Minimize the expected monetary cost of false approvals and false rejections.

• Maximize confirmed fraud captured among the top 200 highest-risk transactions.

This is where machine learning evaluation connects directly to operational design.

8. MAE vs RMSE: evaluating regression errors

Classification metrics measure success at predicting classes. Regression metrics measure the size of numerical prediction errors.

Two of the most widely used regression metrics are mean absolute error and root mean squared error.

Both measure differences between actual and predicted values, but they penalize those differences differently.

Mean absolute error

Mean absolute error calculates the average magnitude of the errors.

MAE = mean(|actual − predicted|)

Suppose a model predicts daily electricity consumption for five households.

Its prediction errors are:

• 2 kWh too low.

• 2 kWh too high.

• 1 kWh too low.

• 1 kWh too high.

• Exactly correct.

The absolute errors are 2, 2, 1, 1, and 0.

MAE = (2 + 2 + 1 + 1 + 0) / 5 = 1.2 kWh

MAE is easy to interpret: across these observations, predictions differ from actual values by 1.2 kWh on average.

Every additional unit of absolute error contributes equally to the total.

An error of 20 contributes twice as much as an error of 10.

Root mean squared error

Root mean squared error takes a different approach.

It squares every error, calculates the mean squared error, and then takes the square root.

RMSE = √(mean((actual − predicted)²))

Using the same five errors:

RMSE = √((4 + 4 + 1 + 1 + 0) / 5)

RMSE = √2 ≈ 1.414 kWh

RMSE is slightly larger than MAE because the squared-error calculation places more weight on the larger errors.

More generally, RMSE is always greater than or equal to MAE when both use the same observations and normalized nonnegative sample weights.

They are equal when all absolute errors have the same magnitude.

Why squaring errors makes a difference

Suppose one prediction is wrong by 2 units.

Its contribution before averaging under the two loss calculations is:

• Absolute error: 2

• Squared error: 4

Now suppose another prediction is wrong by 20 units.

Its contributions are:

• Absolute error: 20

• Squared error: 400

The error increased by a factor of 10, but its squared contribution increased by a factor of 100.

As a result, RMSE is much more sensitive to large individual errors than MAE.

This distinction is crucial for applications where occasional severe prediction mistakes are especially harmful.

9. Runnable example: one outlier changes everything

Let’s demonstrate this difference with actual Python code.

We’ll use an energy consumption forecasting example.

The actual electricity usage is 100 kWh for each of five observations. Our model initially produces small errors.

Then we’ll replace one accurate prediction with a much larger error.

import numpy as np

from sklearn.metrics import (
    mean_absolute_error,
    root_mean_squared_error,
)

# Actual energy consumption in kWh.
y_true = np.array([100, 100, 100, 100, 100])

# Predictions with small errors.
normal_predictions = np.array([
    98, 102, 99, 101, 100
])

# Replace one correct prediction with an outlier.
outlier_predictions = np.array([
    98, 102, 99, 101, 140
])

def evaluate_regression(name, predictions):
    mae = mean_absolute_error(y_true, predictions)
    rmse = root_mean_squared_error(y_true, predictions)

    print(f"\n{name}")
    print(f"MAE:  {mae:.3f} kWh")
    print(f"RMSE: {rmse:.3f} kWh")

evaluate_regression(
    "Normal predictions",
    normal_predictions,
)

evaluate_regression(
    "Predictions with outlier",
    outlier_predictions,
)

Results:

Notice what happened.

Four out of five predictions did not change.

Only the final prediction changed from 100 kWh to 140 kWh, introducing a 40 kWh error.

Yet MAE increased from 1.2 to 9.2, while RMSE increased from approximately 1.414 to 17.944.

Both metrics identify that the model became worse.

But RMSE emphasizes the exceptional mistake far more strongly.

When should you use MAE?

MAE is often a good choice when:

• Prediction errors have approximately linear costs.

• Interpretability matters.

• You want a measure less dominated by extreme residuals.

• Occasional large deviations should not completely overwhelm the typical error level.

Consider a delivery-time prediction service.

If being wrong by 10 minutes is roughly twice as inconvenient as being wrong by 5 minutes, MAE provides a natural summary.

When should you use RMSE?

RMSE is useful when:

• Large errors are disproportionately expensive.

• You want a metric that strongly penalizes severe mistakes.

• Underestimating or overestimating demand by a large amount can create substantial operational problems.

• The modeling objective is closely related to squared-error loss.

For example, consider an energy forecasting system used to plan infrastructure capacity.

If a large forecasting error can create a severe operational disruption, RMSE may align better with the business objective than MAE.

However, RMSE is not automatically better just because it penalizes outliers more heavily.

In real datasets, outliers can result from:

• Legitimate but rare events.

• Incorrect labels.

• Instrumentation failures.

• Data-processing errors.

• Changes in the underlying system.

Before choosing RMSE, investigate whether those extreme observations represent meaningful behavior or bad data.

You should not discard difficult observations merely to improve a metric.

The current scikit-learn API provides root_mean_squared_error directly. Use this rather than the removed squared=False argument of mean_squared_error.

10. MAE and RMSE do not tell the whole regression story

Even a regression model with low MAE and RMSE may be unsuitable for deployment.

Suppose an electricity forecasting model has MAE = 5 kWh.

Is that good?

It depends on the scale of the problem.

If typical consumption is 1,000 kWh, an average error of 5 kWh might be excellent.

If typical consumption is 8 kWh, the same average error could be unacceptable.

Metrics need context.

Compare against a baseline

A useful regression baseline might predict:

• The training-set mean of the target.

• The training-set median of the target.

• Yesterday’s value for a daily forecasting task.

• The value observed at the same time last week.

If a complex neural network achieves MAE = 8 kWh while a simple seasonal baseline achieves MAE = 6 kWh, the sophisticated model has not demonstrated its value.

The right question is not merely whether an error seems small.

It is whether your model improves meaningfully over a reasonable alternative.

Report more than one view of error

For a production forecasting system, consider reporting:

• MAE: average absolute error.

• RMSE: sensitivity to severe errors.

• Median absolute error: the middle absolute error, less influenced by extreme observations.

• 90th or 95th percentile absolute error: how bad errors can be for a substantial minority of cases.

• Mean signed error: whether predictions systematically run too high or too low.

Two models can have the same MAE but very different error distributions.

One might make small mistakes consistently. Another might be nearly perfect most of the time but occasionally produce catastrophic predictions.

The business context determines which behavior is preferable.

Be careful with percentage-error metrics such as MAPE when actual targets are zero or close to zero. Percentage errors can become undefined mathematically or extremely large, making direct comparisons misleading.

11. Metric chooser: what to use for each problem

Here is a practical reference for common supervised machine learning tasks.

This table is a starting point, not a universal rule.

For some applications, the best primary metric is not one of the standard functions in scikit-learn.

A delivery service, for example, might care more about underestimating delivery times than overestimating them.

A fraud-detection system might care far more about missing a high-value fraudulent transaction than missing a low-value one.

In such cases, a custom loss or business-utility function can be more meaningful.

12. The cherry on the cake: a real fraud system overwhelmed by false alarms

The 99%-accuracy example earlier was deliberately synthetic. But the underlying problem is not theoretical.

A particularly revealing real-world example comes from a 2026 case study involving OVEX, a South African cryptocurrency exchange, and machine learning provider DataProphet.

The case study describes a cryptocurrency exchange whose existing production fraud-detection rules generated an enormous number of incorrect alerts.

According to the provider’s published results, the legacy rules caught approximately 75% of confirmed fraud cases.

That sounds reasonably successful until you examine the alerts themselves.

Of the accounts flagged by the existing rules, 93.6% were reportedly legitimate.

This corresponds to precision of approximately 6.4%: only about six genuine fraud cases among every 100 flagged accounts.

The provider reports that its replacement ensemble improved performance considerably.

These are vendor-reported test-set results against the production-rule baseline, not independently verified live-deployment outcomes. The case study does not report that the system had 99% accuracy or a 1% positive-class prevalence.

That distinction matters.

The case demonstrates a real operational failure that accuracy alone cannot expose: a system can generate a huge investigation workload while still missing significant fraud.

It also highlights an often overlooked terminology issue.

False-positive rate and fraction of alerts that are false are not the same metric.

False-positive rate = FP / (FP + TN)

Fraction of alerts that are false = FP / (FP + TP)

The second quantity is also called the false discovery rate and equals 1 − Precision.

Confusing these two measures can make an operational report misleading.

What the same problem looks like at 1% prevalence

Now return to our controlled scenario with exactly 1% fraud.

Suppose a fraud detector achieves:

• Recall = 90%.

• False-positive rate = 1%.

• Positive-class prevalence = 1%.

Out of 10,000 transactions, we would expect:

• 90 correctly detected fraudulent transactions.

• 10 missed fraudulent transactions.

• 99 falsely flagged legitimate transactions.

• 9,801 correctly accepted legitimate transactions.

The resulting precision is:

Precision = 90 / (90 + 99) ≈ 47.6%

Despite catching 90% of all fraud and incorrectly flagging only 1% of legitimate transactions, more than half the alerts would be false.

The accuracy, meanwhile, would be:

Accuracy = (9,801 + 90) / 10,000 = 98.91%

That is a compelling demonstration of why a metric that looks excellent in isolation can become much less impressive when converted into actual operational workload.

The synthetic 1% calculation and the OVEX case are separate examples; they should not be confused or presented as measurements from the same system.

The practical lesson is the same in both: for rare-event detection, evaluate what happens to the positive cases and the alert queue, not just the proportion of all predictions that are correct.

13. Common metric evaluation mistakes to avoid

Even if you select the right metric, the evaluation procedure can still produce misleading results.

Here are several mistakes worth checking for.

Mistake 1: calculating ROC-AUC from hard class predictions

ROC-AUC is intended to evaluate ranking scores across thresholds.

Passing only final 0/1 predictions removes most of the ranking information. A numerical result may still be returned, but it will not provide the comprehensive threshold-independent evaluation usually intended by ROC-AUC.

Use continuous positive-class scores whenever possible.

Mistake 2: optimizing exclusively on the test set

A test set is supposed to estimate performance on unseen data.

If you repeatedly use it to choose models, hyperparameters, features, or thresholds, you are indirectly fitting your development process to that test set.

Use cross-validation or a dedicated validation set for development, and reserve the final test set for independent evaluation.

Mistake 3: reporting only a single score

A fraud system with an F1 score of 0.8 can still have an unacceptable number of missed high-value fraud cases.

A delivery-time model with MAE = 4 minutes can still produce occasional 90-minute errors.

Always examine the underlying errors and the consequences they create.

Mistake 4: evaluating on an unrealistic class distribution

Suppose you rebalance a fraud dataset so that half its examples are fraudulent.

A model evaluated on this artificially balanced dataset may produce precision values that do not resemble its likely production performance.

This happens because precision depends on class prevalence.

Resampling can be useful during training, but final evaluation should reflect the deployment population as closely as possible.

Apply training-data resampling only within the relevant training folds. Never rebalance the held-out test set merely to improve reported metrics.

Mistake 5: ignoring data leakage

A model can achieve outstanding scores by using information that would not exist when predictions must be made.

Examples include:

• Using a chargeback outcome recorded after the transaction.

• Predicting delivery time using a feature created after delivery.

• Forecasting future energy demand using aggregates containing future observations.

• Randomly splitting repeated observations from the same entity when the deployment task requires generalization to new entities.

Leakage can inflate every metric, including ROC-AUC, F1, MAE, and RMSE.

Mistake 6: ignoring temporal drift

Production data changes.

Fraud patterns evolve. Customer behavior shifts. Energy demand changes with seasons, weather, and economic conditions.

A classifier that performs well on randomly shuffled historical data may struggle with transactions from a later period.

For time-dependent applications, evaluate on later data that simulates the actual deployment process. Report performance across relevant time periods rather than relying only on a single aggregate score.

Mistake 7: treating the highest metric as automatically best

Suppose two models have:

Model A has a slightly better F1 score.

But Model B may be preferable for a real-time system if the extra latency or complexity of Model A is unacceptable.

Evaluation metrics guide decisions; they do not replace engineering judgment.

14. A practical evaluation checklist

Before accepting a machine learning model, ask the following questions.

• What exactly are we predicting? A category, a numerical value, or a risk ranking?

• What is the class distribution? Are positive and negative examples similarly common?

• What happens when the model is wrong? Do false positives and false negatives have different costs?

• Does the metric match the decision? Are we ranking observations, triggering alerts, or forecasting quantities?

• Are we using an independent evaluation set? Has the test set influenced model or threshold selection?

• Have we compared against a simple baseline? Does the new model improve meaningfully over a rule or naive prediction?

• Have we inspected failure cases? Which classes, time periods, or value ranges create the worst errors?

• Will the metric remain meaningful in production? What happens if class prevalence, data quality, or operating conditions change?

• Can stakeholders interpret the result? Does a score translate into investigations, money, delay, energy, or another understandable outcome?

If you cannot answer these questions, a high validation score should not be treated as sufficient evidence that the model is ready for deployment.

15. Final takeaways

The most important lesson is simple: the best evaluation metric is the one that reflects the problem you’re actually trying to solve.

Accuracy is useful when correct and incorrect classifications have relatively uniform importance, but it can become dangerously misleading when one class dominates the dataset.

Precision measures whether positive predictions can be trusted.

Recall measures how many true positive cases a model successfully identifies.

F1 balances those two goals, but depends on a chosen threshold and does not directly encode operational costs.

ROC-AUC measures the ability to rank positive cases above negative ones. It can reveal a strong underlying model even when its default classification threshold is poorly chosen.

Average precision is often particularly valuable for evaluating ranking quality in rare-event classification.

For regression, MAE provides a directly interpretable measure of average absolute prediction error, while RMSE puts greater emphasis on large mistakes.

Neither metric is universally superior. Their usefulness depends on whether typical errors or severe errors matter more to the application.

Above all, remember that a metric is a compressed description of model behavior, not a complete description of model quality.

Put it into practice

Create a small evaluation notebook or Python project using the examples in this tutorial. Then run three experiments:

1. Change the fraud prevalence from 1% to 5% and compare accuracy, precision, recall, and F1.

2. Experiment with classification thresholds and observe how F1 changes while ROC-AUC for the same ranking scores stays constant.

3. Increase the size of the regression outlier from 40 kWh to 100 kWh and compare the resulting MAE and RMSE.

For an extra challenge, define a business cost for each false positive and false negative, then find a decision threshold that minimizes that cost rather than maximizing F1.

Your next step: take one model you’re currently evaluating, calculate at least two complementary metrics, inspect its actual errors, and ask whether your chosen metric rewards the behavior your application really needs.

That shift—from optimizing a score to measuring the outcome that matters—is what turns model evaluation into effective machine learning engineering.