Imbalanced classes: why 99% accuracy can mean your model is useless

Imagine building a fraud detection system that achieves 99% accuracy. Your dashboard looks excellent. Your stakeholders are impressed. The model passes testing and gets deployed.

Then you discover something uncomfortable: it has never flagged a single fraudulent transaction.

Every prediction says “legitimate.”

The model is technically 99% accurate because 99% of transactions really are legitimate. But the remaining 1%—the transactions you built the system to catch—are completely invisible to it.

This is the accuracy paradox, one of the most expensive mistakes in classification machine learning.

The problem isn’t necessarily your algorithm. It’s how you’re measuring success.

In this tutorial, we’ll build a reproducible fraud-style classifier, demonstrate exactly why accuracy can be misleading, and compare practical remedies using modern Python tooling.

By the end, you’ll understand:

• Why a classifier can achieve 99% accuracy while detecting zero positive cases.

• How confusion matrices reveal what accuracy hides.

• When precision, recall, F1, and PR-AUC are useful.

• How to apply class weights and resampling correctly.

• Why decision thresholds matter as much as model selection.

• How to evaluate rare-event models against real operational constraints.

We’ll use a completely synthetic transaction dataset, with no downloads or sensitive records required.

1. Understanding class imbalance

A classification dataset is imbalanced when the target classes appear at substantially different frequencies.

In ordinary binary classification, a dataset might have approximately equal numbers of positive and negative examples.

In fraud detection, the distribution could look very different:

The fraudulent transactions form the minority class, while legitimate transactions form the majority class.

This pattern appears across many applications:

• Fraud detection: fraudulent payments may represent a small fraction of transactions.

• Cybersecurity: malicious connections may be rare compared with normal traffic.

• Predictive maintenance: equipment failures may occur infrequently.

• Customer retention: customers who cancel may be less common than customers who remain.

• Manufacturing: defective products may represent a small percentage of output.

Class imbalance is not inherently a problem. It becomes a problem when training objectives, evaluation metrics, or deployment decisions fail to reflect what matters.

A model that gets the majority class right while ignoring the minority can look successful under one metric and fail completely under another.

The 99% accuracy trap

Consider a model with one simple prediction rule:

Every transaction is legitimate.

On our hypothetical dataset of 100,000 transactions, it produces:

• 99,000 correct predictions.

• 1,000 incorrect predictions.

• Zero detected fraud cases.

Accuracy is calculated as:

Accuracy = Correct predictions / Total predictions

Therefore:

Accuracy = 99,000 / 100,000 = 0.99 = 99%

That sounds impressive until we ask the question the fraud team actually cares about:

How much fraud did we detect?

The answer is zero.

This distinction is critical: accuracy measures the proportion of correct classifications, not whether the classifier succeeds at the task that justifies its existence.

2. The confusion matrix: four numbers that tell the real story

Before choosing metrics, understand the confusion matrix.

Binary classification produces four possible outcomes.

Each outcome has a different operational meaning.

True positive (TP): A fraudulent transaction is correctly flagged.

True negative (TN): A legitimate transaction is correctly accepted.

False positive (FP): A legitimate transaction is incorrectly flagged, potentially inconveniencing customers or consuming investigator time.

False negative (FN): A fraudulent transaction goes undetected.

For the always-legitimate classifier, the confusion matrix is:

The model has 99,000 true negatives but zero true positives.

Accuracy is dominated by its correct classifications of the majority class.

The confusion matrix makes the failure immediately visible.

This is why checking the confusion matrix should be a routine part of binary-classifier evaluation—not something reserved for debugging.

3. Precision and recall: the metrics accuracy hides

Two metrics are particularly important when positive events are rare: precision and recall.

Precision: how trustworthy are the alerts?

Precision answers:

Of all transactions flagged as fraudulent, how many actually were fraudulent?

The formula is:

Precision = TP / (TP + FP)

Suppose a model flags 200 transactions, of which 50 are fraudulent.

Precision = 50 / 200 = 25%.

That means investigators checking 200 alerts would find 50 genuine fraud cases and 150 false alarms.

Higher precision generally means fewer wasted investigations per successful detection.

For a classifier that never issues a positive prediction, precision is mathematically undefined because TP + FP = 0. Reporting libraries often assign zero using an explicit zero-division policy.

Recall: how much fraud did we catch?

Recall answers:

Of all fraudulent transactions, how many did the model successfully detect?

The formula is:

Recall = TP / (TP + FN)

Suppose there are 100 fraudulent transactions and the classifier catches 80.

Recall = 80 / 100 = 80%.

The remaining 20 are false negatives.

A classifier that always predicts legitimate has recall equal to zero, regardless of its accuracy.

Why precision and recall compete

Typically, changing the decision threshold creates a tradeoff.

Lowering the threshold causes the classifier to flag more transactions.

This often:

• Increases recall by catching additional fraud cases.

• Decreases precision because more legitimate transactions get flagged.

Increasing the threshold generally does the opposite.

The right balance depends on the operational objective. A manual investigation team handling thousands of transactions cannot investigate unlimited alerts, even if a lower threshold catches more fraud.

F1 and F2 scores

F1 combines precision and recall using their harmonic mean:

F1 = 2 × Precision × Recall / (Precision + Recall)

Unlike accuracy, F1 cannot be high if either precision or recall is very low.

But F1 gives them equal importance.

If detecting additional fraud is more valuable than avoiding some extra investigations, an F-beta score may be more suitable.

For example:

F2 = 5 × Precision × Recall / (4 × Precision + Recall)

F2 places more emphasis on recall.

Neither F1 nor F2 is universally best. They are convenient summaries, not substitutes for a clearly defined business objective.

4. Why PR-AUC is especially useful for rare events

Precision and recall at one threshold only describe one operating point.

What if we want to evaluate a model across many possible thresholds?

A precision-recall curve shows how the two metrics change as the decision threshold changes.

Illustrative precision-recall tradeoff

Conceptual illustration, not measured model results. The teal line shows a possible precision-recall curve; the dashed line represents approximately 1% positive prevalence.A good curve maintains useful precision as recall increases.

PR-AUC versus ROC-AUC

ROC-AUC summarizes the relationship between true-positive rate and false-positive rate.

It is useful for assessing discrimination, but the false-positive rate is calculated against all actual negatives:

FPR = FP / (FP + TN)

When negatives are extremely numerous, a small false-positive rate can still create a large absolute number of false alarms.

For instance, with 990,000 legitimate transactions, a 1% false-positive rate produces 9,900 false alerts.

That’s a substantial operational burden even though the rate appears small.

Precision directly exposes how many positive predictions are genuine, making precision-recall analysis particularly useful in such applications.

A detail about average precision

In scikit-learn, the standard summary is average_precision_score, commonly abbreviated AP.

AP summarizes precision across changes in recall. It is not numerically identical to trapezoidal integration of a precision-recall curve.

The current scikit-learn documentation explicitly distinguishes these computations.

We’ll use AP throughout this tutorial and label it accordingly, rather than treating every PR-AUC calculation as interchangeable.

Another useful reference point: with uninformative random scores, expected average precision is approximately the positive-class prevalence for a sufficiently large dataset. A constant-score classifier also produces AP equal to prevalence.

For our 1%-fraud dataset, that means an AP around 0.01 is essentially a no-ranking-skill baseline.

5. Build a runnable fraud-style experiment

Now we’ll compare four approaches on exactly the same data:

1. A majority-only dummy classifier.

2. Ordinary logistic regression.

3. Logistic regression with class weighting.

4. Logistic regression with random oversampling.

Our synthetic transactions have 12 numerical features and an approximately 1% fraud rate.

We deliberately avoid a real banking dataset so the example runs without credentials, licenses, or external downloads.

Install the tools

Use a Python 3.11-or-newer environment and create a virtual environment.

python -m venv .venv

Activate it on macOS or Linux:

source .venv/bin/activate

Or on Windows PowerShell:

.venv\Scripts\Activate.ps1

Install the libraries:

python -m pip install -U numpy scikit-learn imbalanced-learn

The walkthrough uses maintained APIs supported in the current scikit-learn and imbalanced-learn documentation, including class_weight="balanced", RandomOverSampler, and the imbalanced-learn pipeline.

Create the dataset and split it correctly

Save the following code in a file named fraud_imbalance.py. Subsequent Python blocks in this tutorial should be appended to this file in order.

import numpy as np

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(
    n_samples=40_000,
    n_features=12,
    n_informative=5,
    n_redundant=2,
    n_clusters_per_class=2,
    weights=[0.99, 0.01],
    class_sep=0.8,
    flip_y=0,
    random_state=42,
)

X_trainval, X_test, y_trainval, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    stratify=y,
    random_state=42,
)

X_train, X_val, y_train, y_val = train_test_split(
    X_trainval,
    y_trainval,
    test_size=0.25,
    stratify=y_trainval,
    random_state=42,
)

print(
    "Train, validation, test:",
    len(y_train), len(y_val), len(y_test)
)

print(
    "Positive counts:",
    y_train.sum(), y_val.sum(), y_test.sum()
)

Expected output:

Train, validation, test: 24000 8000 8000
Positive counts: 240 80 80

The split matters more than it might initially appear.

We have three partitions:

• Training: 24,000 records used to fit models.

• Validation: 8,000 records used to choose a decision threshold.

• Test: 8,000 records reserved for final evaluation.

The stratify argument preserves the class proportions across partitions.

This is especially important with small minority classes, where an unstratified random split might accidentally place very few—or even no—positive cases in one partition.

For real transaction systems, a chronological split may be more appropriate than a random split. Fraud patterns evolve, and evaluating historical transactions using information from the future can produce misleading results.

For this stationary synthetic demonstration, a stratified random split keeps the experiment simple.

6. Compare the four classifiers

First, build the models.

from imblearn.over_sampling import RandomOverSampler
from imblearn.pipeline import Pipeline as ImbPipeline

from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

models = {
    "Majority-only": DummyClassifier(
        strategy="most_frequent"
    ),

    "Ordinary logistic": make_pipeline(
        StandardScaler(),
        LogisticRegression(max_iter=1000),
    ),

    "Weighted logistic": make_pipeline(
        StandardScaler(),
        LogisticRegression(
            class_weight="balanced",
            max_iter=1000,
        ),
    ),

    "Oversampled logistic": ImbPipeline([
        ("scale", StandardScaler()),
        (
            "sample",
            RandomOverSampler(random_state=42),
        ),
        ("model", LogisticRegression(max_iter=1000)),
    ]),
}

The dummy classifier represents our intentionally unhelpful baseline.

Ordinary logistic regression learns from the natural class distribution.

The weighted classifier increases the training influence of rare positive examples.

The oversampled classifier duplicates minority-class training examples until the classes are balanced.

All three logistic regression variants scale their features.

Notice the sampler belongs inside an imbalanced-learn pipeline rather than a regular scikit-learn pipeline.

We’ll explain why shortly.

Evaluate predictions and probabilities

Next, create an evaluation function.

from sklearn.metrics import (
    accuracy_score,
    average_precision_score,
    confusion_matrix,
    precision_score,
    recall_score,
)

def evaluate(y_true, scores, threshold=0.50):
    predicted = (scores >= threshold).astype(int)

    tn, fp, fn, tp = confusion_matrix(
        y_true,
        predicted,
        labels=[0, 1],
    ).ravel()

    return {
        "accuracy": accuracy_score(y_true, predicted),
        "precision": precision_score(
            y_true, predicted, zero_division=0
        ),
        "recall": recall_score(
            y_true, predicted, zero_division=0
        ),
        "ap": average_precision_score(y_true, scores),
        "alerts": int(predicted.sum()),
        "tp": int(tp),
        "fp": int(fp),
        "fn": int(fn),
        "tn": int(tn),
    }


for name, model in models.items():
    model.fit(X_train, y_train)

    scores = model.predict_proba(X_test)[:, 1]
    metrics = evaluate(y_test, scores)

    print(
        f"{name:21} "
        f"accuracy={metrics['accuracy']:.3f} "
        f"precision={metrics['precision']:.3f} "
        f"recall={metrics['recall']:.3f} "
        f"AP={metrics['ap']:.3f} "
        f"alerts={metrics['alerts']} "
        f"TP={metrics['tp']} "
        f"FP={metrics['fp']} "
        f"FN={metrics['fn']} "
        f"TN={metrics['tn']}"
    )

Run the complete file so far:

python fraud_imbalance.py

The results below were generated from this seeded experiment using scikit-learn 1.8.0 and imbalanced-learn 0.14.1. Compatible library versions should produce broadly similar results, though exact numbers can vary.

Results: accuracy versus what actually matters

These results expose several important lessons.

Lesson 1: identical accuracy can hide radically different models

The majority-only model and ordinary logistic regression both achieve approximately 99% accuracy.

But ordinary logistic regression detects five actual fraud cases.

The majority-only model catches none.

Even when accuracy is virtually identical, the models are not functionally equivalent.

Lesson 2: better recall can dramatically reduce accuracy

The weighted classifier detects 67 of the 80 fraudulent transactions.

Its recall is approximately 83.8%.

That’s a massive improvement over the ordinary model’s 6.2%.

But its accuracy falls to approximately 82%.

Why?

Look at the confusion matrix.

The weighted model raises 1,495 alerts, of which only 67 are genuine fraud cases.

That’s 4.5% precision.

If each false alarm requires manual investigation, this might be operationally expensive.

The highest-recall classifier is not automatically the best classifier.

Lesson 3: changing class weights can also affect ranking quality

Notice something surprising.

Ordinary logistic regression achieves an AP of 0.193, while weighted logistic regression achieves 0.123.

The weighted model catches many more fraud cases at the default threshold, yet the ordinary model produces a better overall precision-recall ranking in this experiment.

There is no contradiction.

Recall at a particular threshold and average precision across thresholds measure different aspects of performance.

Class weighting changes the optimization objective. It does not guarantee improved ranking quality.

This is a key reason to compare multiple metrics and thresholds before selecting a model.

7. Class weights: a simple and powerful first experiment

Class weighting modifies how the learning algorithm penalizes errors.

Without weighting, each training example generally contributes according to the model’s standard loss function.

With balanced class weights, errors on rare examples receive greater influence.

For binary classes, scikit-learn’s balanced weighting uses:

Class weight = Total training samples / (Number of classes × Samples in that class)

With 24,000 training examples, 240 positives, and 23,760 negatives:

• Positive-class weight = 24,000 / (2 × 240) = 50.

• Negative-class weight = 24,000 / (2 × 23,760) ≈ 0.505.

A positive example therefore receives almost 99 times the weight of a negative example.

This is useful when the minority class would otherwise contribute too little to the model’s objective.

Advantages include:

• No synthetic observations.

• No discarded majority examples.

• Simple integration with supported estimators.

• Low additional memory requirements.

However, class weighting has limitations.

It may increase false alarms substantially, and a class-weighted model’s raw probability outputs should not automatically be treated as calibrated real-world event probabilities.

Class weighting changes training behavior; it does not change the true prevalence of fraud.

Use it as a candidate strategy, not as an automatic production setting.

8. Resampling: changing what the model sees during training

Resampling modifies the training distribution to make minority examples more influential.

There are three common approaches.

Random oversampling

Random oversampling duplicates existing minority-class examples, typically sampling with replacement.

Suppose the training data contains:

• 23,760 legitimate transactions.

• 240 fraudulent transactions.

Full random oversampling produces:

• 23,760 legitimate examples.

• 23,760 positive examples, including duplicates.

Unlike class weighting, this changes the number of training rows.

Oversampling can help some algorithms pay more attention to rare examples, but repeated records introduce no new independent information and can encourage overfitting.

Random undersampling

Random undersampling removes some majority-class examples.

For example, one might reduce 23,760 legitimate examples to 2,400 while preserving all 240 fraudulent examples.

The resulting ratio is 10:1 instead of 99:1.

Benefits include reduced training cost and a less extreme class distribution.

The major disadvantage is information loss.

Discarded legitimate transactions might contain useful patterns needed to distinguish unusual but legitimate behavior from actual fraud.

Synthetic oversampling

SMOTE—Synthetic Minority Over-sampling Technique—creates synthetic minority-class observations by interpolating between nearby examples.

It can be useful when the feature space supports meaningful interpolation.

However, synthetic combinations are not automatically realistic transactions.

In particular, ordinary SMOTE is not a drop-in solution for arbitrary categorical identifiers, constrained features, or temporal relationships.

For such data, domain-aware processing and compatible samplers are essential.

The current imbalanced-learn guide documents random oversampling, undersampling, SMOTE, and appropriate variants for different feature types.

The most dangerous resampling mistake

Never resample the entire dataset before splitting it.

Consider random oversampling.

If you duplicate positive records first and then randomly create training and test sets, exact copies of the same observation may appear in both sets.

The model can effectively encounter test examples during training.

With SMOTE, synthetic samples can similarly incorporate information from examples that later appear in validation folds.

The resulting evaluation can become unrealistically optimistic.

The correct process is:

1. Split the original dataset.

2. Preserve the natural class distribution in validation and test sets.

3. Apply resampling only to training data.

4. During cross-validation, resample independently inside each training fold.

The imbalanced-learn pipeline handles this correctly: samplers run during fitting, rather than resampling validation or prediction inputs. Its official guidance explicitly warns against resampling before data splitting.

Our example follows this pattern.

9. Decision thresholds: the missing part of many ML tutorials

Most introductory classification workflows train a model and call its prediction method.

For binary probabilistic classifiers, this usually means selecting the positive class when its predicted probability reaches 0.5.

But 0.5 is not automatically the right threshold for a rare-event system.

Suppose a transaction receives a fraud score of 0.18.

At threshold 0.5, the classifier predicts legitimate.

At threshold 0.10, it predicts fraud.

Nothing about the trained model has changed.

Only the decision policy has changed.

This separation is useful because model training, ranking quality, and operational decisions are distinct problems.

Choose a threshold using validation data

Let’s tune the weighted classifier to maximize F2 on our validation partition.

Append this code to the same Python file:

from sklearn.metrics import precision_recall_curve

weighted = models["Weighted logistic"]

validation_scores = weighted.predict_proba(X_val)[:, 1]

precision, recall, thresholds = precision_recall_curve(
    y_val,
    validation_scores,
)

beta_sq = 4.0

f2 = (
    (1 + beta_sq) * precision[:-1] * recall[:-1]
    / np.maximum(
        beta_sq * precision[:-1] + recall[:-1],
        1e-12,
    )
)

best_index = int(np.argmax(f2))
selected_threshold = float(thresholds[best_index])

print(
    f"Validation-selected threshold (max F2): "
    f"{selected_threshold:.3f}"
)

weighted_test_scores = weighted.predict_proba(X_test)[:, 1]

chosen = evaluate(
    y_test,
    weighted_test_scores,
    threshold=selected_threshold,
)

print("Weighted logistic, tuned:", chosen)

Notice that precision_recall_curve returns one more precision and recall value than thresholds. The final endpoint has no corresponding threshold, which is why the calculation uses [:-1].

The selected threshold in this experiment is approximately 0.809.

That is higher than 0.5 because our validation objective penalizes very low precision as well as missed cases.

What happens on the held-out test set?

The new threshold produces fewer alerts and higher precision, but it also detects fewer fraud cases.

Neither threshold is universally correct.

The first catches more fraud while demanding substantially more investigation capacity.

The second reduces false alarms but misses additional fraudulent transactions.

For a real deployment, an F2-optimal threshold should be treated as a candidate rather than a final answer. It reflects a mathematical preference, not necessarily the true costs and capacities of the organization.

Also remember that our validation set contains only 80 positive cases. Threshold estimates based on relatively few positives can be unstable. Repeated evaluation, confidence intervals, and larger representative validation windows may be needed.

10. Evaluate under a real alert budget

Suppose the fraud operations team can inspect only 80 transactions out of each batch of 8,000.

In this scenario, the team cannot act on every positive prediction from the weighted classifier.

A fixed decision threshold that generates hundreds or thousands of alerts does not fit the workflow.

Instead, the system might sort transactions by fraud score and send only the highest-ranked 80 to investigation.

This is a precision-at-k problem.

Precision at k asks:

Among the k highest-ranked transactions, what proportion are actually fraudulent?

Recall at k measures the fraction of all fraudulent transactions captured in those k slots.

Let’s implement it.

capacity = 80

for name in [
    "Ordinary logistic",
    "Weighted logistic",
    "Oversampled logistic",
]:
    scores = models[name].predict_proba(X_test)[:, 1]

    top_indices = np.argsort(
        -scores,
        kind="stable",
    )[:capacity]

    detected = int(y_test[top_indices].sum())

    precision_at_k = detected / capacity
    recall_at_k = detected / int(y_test.sum())

    print(
        f"{name}: "
        f"caught={detected}, "
        f"precision@{capacity}={precision_at_k:.3f}, "
        f"recall@{capacity}={recall_at_k:.3f}"
    )

Results:

This result is particularly instructive.

The ordinary logistic model, which appeared weak at the default classification threshold, produces more genuine fraud cases in the top-80 queue.

Its ranking quality is better for this particular capacity constraint.

This does not prove that unweighted models are generally superior. It demonstrates that changing the decision problem can change which model is most useful.

If the operational objective is to investigate the highest-risk transactions with limited staff, ranking metrics may be more relevant than recall at a fixed threshold.

In practice, select the model and queue policy using validation data or appropriate cross-validation. Use the untouched test data only for final assessment. We’ve examined multiple test results here for teaching purposes; repeated model selection based on these results would invalidate the test set as an unbiased final evaluation.

11. Translate model performance into business consequences

Statistical metrics become easier to interpret when connected to operational costs.

Suppose:

• Every false alarm (a legitimate transaction sent to review) wastes $4 of analyst time. Reviewing a genuine fraud case is part of the prevention benefit, so it is not counted as a cost.

• Every missed fraudulent transaction produces an average expected loss of $200.

• Every correctly detected fraudulent transaction is prevented before the loss occurs.

These are hypothetical assumptions. A real business model would need to account for recoveries, investigation outcomes, transaction values, customer friction, and other effects.

A simple loss approximation is:

Total cost = (FP × $4) + (FN × $200)

Using our test-set confusion matrices:

The high-recall weighted model looks much better under these particular assumptions despite its lower accuracy.

But change the assumed cost of investigating a false positive, and the ordering may change.

This leads to a general rule:

The ideal model is the one that supports the best feasible decisions under the costs, constraints, and risks of the real application—not necessarily the one with the highest standalone metric.

The cost model must also be validated. Treating average missed-fraud losses as identical across every case is often unrealistic.

For example, a model detecting many small fraudulent payments may have higher recall but prevent less monetary loss than one detecting fewer, much larger fraudulent transactions.

Weighted financial recall or expected prevented loss might then be more appropriate.

12. Cherry on the cake: what happens when high accuracy reaches production?

The risk of attractive offline metrics failing to translate into useful outcomes is not merely theoretical.

In a first-person account published on Medium in 2026, “Our Classifier Was 91% Accurate — Until the Data Had Memory”, the author described a customer-churn classifier that achieved 91.3% test accuracy and was deployed.

According to the account, the team subsequently found that the model had missed 60% of the customers who actually churned.

The reported issue was not solely class imbalance. The model relied on snapshot-style features while the relevant customer behavior unfolded over time. The account is self-reported rather than an independently audited incident.

That distinction matters: the author did not report that the deployed classifier literally predicted the majority class for everyone.

Our majority-only experiment is the more extreme, fully reproducible example of that failure mode.

It achieves 99% accuracy while detecting absolutely nothing.

The production anecdote adds another lesson: even using better metrics will not rescue a model whose features, validation design, or underlying problem definition do not match reality.

A useful predeployment safety check is to compare any classifier against a constant majority-class baseline and verify that the expected positive prediction rate, recall, and alert volume are meaningful.

If a fraud model predicts every transaction as legitimate, a headline accuracy score should not be enough to approve deployment.

13. A practical checklist before shipping an imbalanced classifier

The techniques in this tutorial are only part of a reliable production workflow.

Before deployment, check the following.

Validate the labels

Rare positives are especially vulnerable to label-quality problems.

Confirm that:

• The target definition matches the business event.

• Negative labels are not merely uninvestigated or unconfirmed transactions.

• Known label delays are handled.

• Cases without reliable outcomes are not silently treated as negatives.

A fraud detector can appear excellent when the training data contains incomplete or biased labels.

Use representative validation

A synthetic stationary dataset is helpful for teaching, but real financial transactions have temporal patterns.

Consider:

• Chronological validation.

• Changes in transaction volume and fraud prevalence.

• Previously unseen accounts, devices, merchants, or campaigns.

• Grouping related records to avoid leakage across partitions.

• Appropriate evaluation windows for delayed fraud confirmation.

Stratification alone cannot protect against these problems.

Compare with simple baselines

Always include a majority-class classifier as a sanity check.

Then compare with something simple and interpretable, such as logistic regression.

Complexity should earn its place through improved performance on the actual operational objective.

Measure more than one quantity

For rare-event binary classification, a useful report normally includes:

• Class prevalence and number of positive test cases.

• Confusion matrix, precision, and recall.

• Average precision and precision-recall curve.

• Alert volume and precision at the operational review capacity.

• Calibration quality when scores are used as probabilities.

• Estimated financial or operational consequences.

Balanced accuracy and ROC-AUC can also add context, but they should not replace these direct operational measurements.

Select thresholds independently of the test set

A disciplined process is:

1. Fit models using training data.

2. Select models and thresholds using validation data or cross-validation.

3. Evaluate once on a held-out test set.

4. If the chosen policy changes, collect fresh evaluation data or preserve another untouched holdout.

Otherwise, repeated experimentation can turn the test set into another tuning set.

Monitor after deployment

Watch for changes in:

• Fraud prevalence.

• Predicted score distributions.

• Alert counts and review queues.

• Precision and recall on newly confirmed labels.

• Feature distributions.

• Actual loss and investigation cost.

Immediate monitoring can track score and alert distributions even before labels arrive. Performance metrics requiring true outcomes must wait for sufficiently reliable labels.

An unusually sharp decline in the number of positive predictions deserves investigation, especially if the model previously detected rare events consistently.

14. Common mistakes and their better alternatives

One subtle issue deserves particular attention: calibration.

A classifier can rank transactions correctly without producing reliable numerical probabilities.

For example, if a model assigns a score of 0.70 to a group of transactions, good calibration would mean that approximately 70% of those transactions are positive in an appropriate representative sample.

Class weighting and oversampling can change the relationship between scores and real event probabilities.

If the downstream system uses scores to estimate expected monetary loss, calibration becomes particularly important.

Calibration itself needs independent validation and must not inadvertently expose the model to held-out test labels.

15. What to remember

The main lesson is not that accuracy is a bad metric.

Accuracy is useful when the distribution of classes, the costs of mistakes, and the intended use make it informative.

The mistake is treating it as a universal measure of classification quality.

From this tutorial, we can draw five practical conclusions.

First, compare against the majority baseline. If a model cannot demonstrate value beyond predicting the common outcome, a high accuracy number means very little.

Second, evaluate the minority class explicitly. Precision and recall reveal what happens to the events the system is supposed to detect.

Third, distinguish threshold quality from ranking quality. A model can perform poorly at threshold 0.5 while still ranking positive cases usefully.

Fourth, treat class weighting and resampling as experiments, not guaranteed improvements. They change model behavior and can create substantial tradeoffs.

Fifth, judge performance within the real workflow. Investigation capacity, costs, label delays, and future changes all affect whether a model creates value.

Our synthetic experiment made this concrete. The model with 99% accuracy and zero recall detected nothing. The weighted model caught far more fraud but generated many false alerts. The unweighted model achieved better average precision and caught more positives in a fixed-size investigation queue.

None of those observations requires a complicated neural network. They require measuring the right things.

16. Your next step: audit one of your own classifiers

Take an existing classification project—or run the synthetic experiment above—and perform this exercise:

• Calculate the majority-class baseline accuracy.

• Produce a confusion matrix for your current model.

• Report positive-class precision, recall, and average precision.

• Compare ordinary training, balanced class weights, and training-only resampling.

• Select a threshold using validation data.

• Evaluate how many useful detections fit within a realistic alert budget.

If you want to go further, repeat the experiment with fraud prevalence reduced from 1% to 0.1%. Observe how precision changes even when the underlying ranking quality remains useful. That exercise makes the base-rate problem especially tangible.

Your challenge: stop asking whether your classifier is 99% accurate. Start asking how many important events it finds, how many false alarms it creates, and whether the people using its predictions can act on them.

That’s the difference between a model that produces an impressive number and a model that delivers measurable value.