Simpson’s paradox: when your aggregate answer is exactly wrong

1. The paradox: when correct calculations lead to the wrong conclusion

Imagine you are a data analyst at an e-commerce company. Your team uses two shipping carriers, ExpressSwift and BudgetFlow, and management wants to know which one delivers packages on time more reliably.

You analyze 2,000 shipments and produce this report:

The conclusion seems obvious: BudgetFlow outperforms ExpressSwift by 23 percentage points.

Your manager proposes moving more shipments to BudgetFlow. The numbers support that decision, right?

Not necessarily.

Before making the recommendation, you separate the shipments into two delivery zones:

• Easy: Urban and suburban destinations with predictable routes, short distances, and reliable infrastructure.

• Hard: Remote or geographically challenging destinations where deliveries are more difficult to complete on time.

Now the results look completely different.

ExpressSwift performs better in both delivery zones, yet BudgetFlow wins when everything is combined.

There is no calculation error. The data is consistent. Every percentage is correct.

The reversal happens because the two carriers handle very different mixes of shipments.

This is Simpson’s paradox: a statistical phenomenon in which a relationship observed within separate groups disappears or reverses when those groups are combined.

It matters because real-world decisions are frequently based on aggregated metrics:

• Which marketing campaign converts better?

• Which shipping company performs more reliably?

• Which website version improves conversion?

• Which sales team closes more deals?

• Which machine-learning model performs better across different customer segments?

An aggregate metric may correctly describe what happened while misleading you about which option performs better under comparable conditions.

In this tutorial, we’ll reproduce the paradox with Python and pandas, investigate the mathematics behind the reversal, correct the comparison using standardization, and examine the famous University of California, Berkeley admissions case using its actual historical numbers.

By the end, you’ll be able to detect the paradox in a dataset, explain precisely why it happens, and avoid some of the most dangerous mistakes in interpreting grouped statistics.

2. Set up the Python environment

We’ll use Python and pandas to build a reproducible dataset, calculate aggregate and grouped statistics, and demonstrate how changing the weighting scheme changes the result.

The tutorial requires:

• Python 3.11 or newer.

• pandas for data manipulation (we tested with pandas 3.0.6 on Python 3.13).

• A terminal and a code editor.

• Basic familiarity with DataFrames and grouping operations.

Create a project directory and virtual environment.

On macOS or Linux:

mkdir simpsons-demo
cd simpsons-demo

python -m venv .venv
source .venv/bin/activate

python -m pip install pandas==3.0.6

We’ll generate the dataset directly in Python. No external download, authentication, or API key is required.

This also makes the example deterministic: every run produces the same number of shipments and exactly the same reversal.

3. Build a realistic delivery dataset

Our dataset represents one completed delivery per row. Each delivery has three attributes:

We’ll construct the records using four groups with known shipment counts.

Notice something important before we even run the code.

ExpressSwift handles 900 difficult deliveries and only 100 easy deliveries. BudgetFlow has almost the opposite workload.

This imbalance is not an incidental detail. It is the mechanism responsible for the paradox.

Create the complete runnable experiment

Create a file named simpsons_demo.py with the following code.

import pandas as pd

# Fixed counts make the experiment fully reproducible.
groups = [
    ("ExpressSwift", "Easy", 100, 95),
    ("ExpressSwift", "Hard", 900, 540),
    ("BudgetFlow", "Easy", 900, 810),
    ("BudgetFlow", "Hard", 100, 55),
]

# Generate individual shipment records.
records = [
    {
        "carrier": carrier,
        "zone": zone,
        "on_time": index < successes,
    }
    for carrier, zone, shipments, successes in groups
    for index in range(shipments)
]

shipments = pd.DataFrame.from_records(records)

# Shuffle records without changing the group totals.
shipments = shipments.sample(
    frac=1,
    random_state=42,
).reset_index(drop=True)

assert len(shipments) == 2000


def summarize(frame, keys):
    summary = (
        frame.groupby(keys, as_index=False)
        .agg(
            shipments=("on_time", "size"),
            on_time_deliveries=("on_time", "sum"),
            on_time_rate=("on_time", "mean"),
        )
        .sort_values(keys)
    )

    summary["on_time_pct"] = 100 * summary["on_time_rate"]

    return summary


# Step 1: Compare carriers without considering zones.
overall = summarize(shipments, ["carrier"])

# Step 2: Compare carriers within each zone.
by_zone = summarize(shipments, ["zone", "carrier"])

print("OVERALL")
print(
    overall[
        [
            "carrier",
            "shipments",
            "on_time_deliveries",
            "on_time_pct",
        ]
    ].to_string(index=False)
)

print("\nBY ZONE")
print(
    by_zone[
        [
            "zone",
            "carrier",
            "shipments",
            "on_time_deliveries",
            "on_time_pct",
        ]
    ].to_string(index=False)
)


# Step 3: Use a shared distribution of delivery zones.
rates = by_zone.pivot(
    index="zone",
    columns="carrier",
    values="on_time_rate",
)

counts = by_zone.groupby("zone")["shipments"].sum()
weights = counts / counts.sum()

standardized = (
    rates.mul(weights, axis=0)
    .sum(axis=0)
    .mul(100)
)

print("\nZONE WEIGHTS")
print(weights.to_string())

print("\nSTANDARDIZED ON-TIME RATES (%)")
print(standardized.to_string())


# Step 4: Verify that Simpson's paradox occurred.
assert rates["ExpressSwift"].gt(
    rates["BudgetFlow"]
).all()

overall_rates = overall.set_index("carrier")["on_time_rate"]

assert (
    overall_rates["ExpressSwift"]
    < overall_rates["BudgetFlow"]
)

assert (
    standardized["ExpressSwift"]
    > standardized["BudgetFlow"]
)

print("\nAll reversal checks passed.")

Run the experiment:

python simpsons_demo.py

The pandas operations use named aggregation, a supported technique for calculating several statistics with clearly labeled output columns.

Understanding the dataset generation

We do not generate random success probabilities and hope for a reversal.

Instead, we specify exactly how many deliveries succeeded in each group.

The expression comparing an index against the number of successes creates that many successful deliveries, followed by unsuccessful deliveries.

For example, the ExpressSwift easy-zone group contains 100 records, of which exactly 95 have a successful delivery outcome.

The subsequent shuffle changes the row order, not the statistical relationships.

This distinction matters in educational experiments. If outcomes were sampled randomly without fixed counts, some runs could produce a weaker reversal or no reversal at all.

Here, the effect is guaranteed, which makes it easier to isolate the mathematics.

4. Calculate the aggregate performance

The first output is:

OVERALL
     carrier  shipments  on_time_deliveries  on_time_pct
  BudgetFlow       1000                 865         86.5
ExpressSwift       1000                 635         63.5

At the aggregate level, BudgetFlow is clearly ahead.

The metric is calculated as:

On-time rate = On-time deliveries / Total deliveries × 100

For ExpressSwift:

635 / 1,000 × 100 = 63.5%

For BudgetFlow:

865 / 1,000 × 100 = 86.5%

The difference is:

86.5 − 63.5 = 23 percentage points.

A conventional dashboard might stop here.

Indeed, if the question were simply which carrier had the higher overall on-time rate across the shipments it actually handled, this would be an accurate answer.

The problem begins when that descriptive result is used to answer a different question: which carrier performs better when handling comparable deliveries?

Those are not equivalent questions.

5. Split the data into meaningful groups

The second output reveals the reversal.

BY ZONE
zone      carrier  shipments  on_time_deliveries  on_time_pct
Easy   BudgetFlow        900                 810         90.0
Easy ExpressSwift        100                  95         95.0
Hard   BudgetFlow        100                  55         55.0
Hard ExpressSwift        900                 540         60.0

Now calculate the differences.

In the easy zone:

• ExpressSwift: 95.0%

• BudgetFlow: 90.0%

• ExpressSwift advantage: 5 percentage points.

In the hard zone:

• ExpressSwift: 60.0%

• BudgetFlow: 55.0%

• ExpressSwift advantage: 5 percentage points.

ExpressSwift has a positive advantage in every zone.

Yet the aggregate comparison shows a negative difference of 23 percentage points.

On-time delivery rate by comparison

ExpressSwiftBudgetFlowSynthetic dataset of 2,000 shipments. Percentage-point differences use ExpressSwift minus BudgetFlow.

This is the central lesson of the experiment.

Aggregating observations changes the effective weighting of their underlying groups.

And those weights can change the direction of the comparison.

6. Why Simpson’s paradox happens: the mathematics

There is nothing mathematically contradictory about the result.

The paradox comes from how weighted averages work.

For any carrier, its overall on-time rate is a weighted average of its within-zone on-time rates.

In plain notation:

Overall rate = Σ (Zone rate × Zone share)

The zone share is the fraction of that carrier’s deliveries belonging to a particular zone.

ExpressSwift’s weighted average

ExpressSwift’s distribution is:

• Easy deliveries: 10%

• Hard deliveries: 90%

Its overall rate is:

Overall = (0.10 × 95%) + (0.90 × 60%)

Overall = 9.5% + 54.0%

Overall = 63.5%

Most of ExpressSwift’s deliveries belong to the harder group, which has a much lower success rate.

BudgetFlow’s weighted average

BudgetFlow’s distribution is:

• Easy deliveries: 90%

• Hard deliveries: 10%

Its overall rate is:

Overall = (0.90 × 90%) + (0.10 × 55%)

Overall = 81.0% + 5.5%

Overall = 86.5%

BudgetFlow’s easier workload raises its aggregate performance substantially.

The hidden variable

The relevant variable is delivery-zone difficulty.

It is associated with both:

1. Which carrier handles a shipment.

2. Whether that shipment arrives on time.

In statistical terminology, this makes zone difficulty a plausible confounding variable for the relationship between carrier and delivery performance.

A confounder can create or obscure an association when it is ignored.

In our constructed example, the carriers’ within-zone differences are identical: ExpressSwift leads by exactly five percentage points in either environment.

The aggregate reversal comes from the differences in their shipment distributions.

Why weighted averages can reverse rankings

Suppose Carrier A performs better than Carrier B in every delivery zone.

If both carriers are evaluated using exactly the same zone weights, A must also have the higher weighted-average performance.

But when each carrier uses its own zone distribution, that guarantee disappears.

The aggregate comparison combines two things:

• Differences in performance within zones.

• Differences in the proportions of shipments assigned to those zones.

The second effect can be strong enough to overwhelm the first.

Our example demonstrates precisely that situation.

7. Correct the comparison with standardization

One useful way to make the comparison more meaningful is direct standardization.

Instead of using each carrier’s actual shipment distribution, we evaluate both carriers against a common distribution of shipment difficulty.

This answers the question:

What would each carrier’s expected on-time rate be if both handled the same mixture of easy and hard deliveries, assuming their within-zone rates remained unchanged?

For this experiment, the entire dataset contains:

• 1,000 easy-zone deliveries.

• 1,000 hard-zone deliveries.

Therefore, the pooled target population consists of 50% easy deliveries and 50% hard deliveries.

These weights are already computed in the script.

ZONE WEIGHTS
zone
Easy    0.5
Hard    0.5

Apply the same weights to both carriers

For ExpressSwift:

Standardized rate = (0.50 × 95%) + (0.50 × 60%)

Standardized rate = 47.5% + 30.0%

Standardized rate = 77.5%

For BudgetFlow:

Standardized rate = (0.50 × 90%) + (0.50 × 55%)

Standardized rate = 45.0% + 27.5%

Standardized rate = 72.5%

The Python output confirms the calculations.

STANDARDIZED ON-TIME RATES (%)
carrier
BudgetFlow      72.5
ExpressSwift    77.5

All reversal checks passed.

How the ranking changes after using shared zone weights

ExpressSwiftBudgetFlowStandardized comparison uses 50% easy and 50% hard deliveries for both carriers.

After standardization, ExpressSwift leads by five percentage points.

This agrees with our within-zone analysis.

What pandas is doing

The relevant part of the original code is the multiplication of subgroup rates by common subgroup weights.

The pivot operation creates a table in which:

• Each row represents a delivery zone.

• Each column represents a carrier.

• Each cell contains that carrier’s on-time rate for that zone.

The zone counts determine the distribution of the pooled population.

Dividing those counts by their total produces weights that sum to one.

The final calculation multiplies every carrier’s rate by the same zone weights, sums the results, and converts the proportions into percentages.

This method is sometimes called adjusted rate comparison or standardized rate comparison.

It is useful in many domains, including product analytics, operational monitoring, and observational research.

However, standardization does not automatically turn an observational comparison into a causal experiment.

8. Choosing the right weights for a real decision

An important practical detail is that the target distribution must match the question being asked.

Our 50/50 weighting scheme is reasonable because it represents the pooled distribution of all shipments in the dataset.

But imagine your business expects next quarter’s orders to consist of:

• 30% easy deliveries.

• 70% hard deliveries.

You can standardize both carriers to that expected workload instead.

Append this code to the original script:

future_weights = pd.Series(
    {
        "Easy": 0.30,
        "Hard": 0.70,
    }
)

assert abs(future_weights.sum() - 1.0) < 1e-12

future_rates = (
    rates.mul(future_weights, axis=0)
    .sum(axis=0)
    .mul(100)
)

print("\nEXPECTED RATES FOR NEXT QUARTER (%)")
print(future_rates.to_string())

The result is:

EXPECTED RATES FOR NEXT QUARTER (%)
carrier
BudgetFlow      65.5
ExpressSwift    70.5

ExpressSwift still leads by five percentage points.

In this synthetic example, that advantage persists under every common mixture of the two zones because its rate is exactly five points higher in both.

Real datasets are often less convenient.

One carrier might perform much better in city centers but slightly worse in remote areas. The preferred carrier would then depend on the expected proportion of shipments in each environment.

For that reason, selecting a target distribution is not just a mathematical step. It is a business decision.

Useful target populations might include:

• The entire customer base.

• The projected workload for the next quarter.

• A representative sample of future website traffic.

• Customers eligible for both alternatives.

• A fixed reference population used consistently across reporting periods.

Different targets can produce different standardized results without any error in the calculations.

The key is to make the target population explicit.

9. The cherry on the cake: the real Berkeley admissions case

One of the most famous historical examples of Simpson’s paradox comes from graduate admissions at the University of California, Berkeley.

In 1973, the university received 12,763 applications across 101 graduate departments and interdepartmental programs.

The aggregated figures seemed to indicate a substantial admissions disparity between men and women.

The difference was approximately 9.7 percentage points in favor of men.

At first glance, these numbers appeared to provide evidence of disadvantage for female applicants in the admissions process.

The case prompted an investigation by researchers Peter J. Bickel, Eugene A. Hammel, and J. W. O’Connell, whose findings were published in Science on February 7, 1975 (“Sex Bias in Graduate Admissions: Data from Berkeley”).

What happened when researchers examined departments?

Admission rates differed dramatically between departments.

Some were extremely selective. Others admitted a much larger proportion of applicants.

Moreover, men and women did not apply to these departments in the same proportions.

Consider the actual data from Berkeley’s six largest departments, conventionally anonymized as A through F.

These figures are available in the documented UCBAdmissions dataset and in statistical teaching materials based on the original study.

Let’s reproduce the comparison with pandas.

Reproduce the departmental analysis

Create a second script named berkeley_check.py.

import pandas as pd

berkeley = pd.DataFrame(
    [
        ("A", 825, 512, 108, 89),
        ("B", 560, 353, 25, 17),
        ("C", 325, 120, 593, 202),
        ("D", 417, 138, 375, 131),
        ("E", 191, 53, 393, 94),
        ("F", 373, 22, 341, 24),
    ],
    columns=[
        "department",
        "men_applied",
        "men_admitted",
        "women_applied",
        "women_admitted",
    ],
)

# Compute admission rates within each department.
for group in ["men", "women"]:
    berkeley[f"{group}_rate"] = (
        100
        * berkeley[f"{group}_admitted"]
        / berkeley[f"{group}_applied"]
    )

print(
    berkeley[
        ["department", "men_rate", "women_rate"]
    ].round(1).to_string(index=False)
)

# Compute totals for these six departments only.
for group in ["men", "women"]:
    applicants = berkeley[f"{group}_applied"].sum()
    admitted = berkeley[f"{group}_admitted"].sum()

    print(
        f"{group}: {admitted}/{applicants} "
        f"= {100 * admitted / applicants:.1f}%"
    )

better_for_women = (
    berkeley["women_rate"] > berkeley["men_rate"]
).sum()

print(
    "Departments where women's admission "
    "rate is higher:",
    int(better_for_women),
)

Run it:

python berkeley_check.py

The output is:

department  men_rate  women_rate
         A      62.1        82.4
         B      63.0        68.0
         C      36.9        34.1
         D      33.1        34.9
         E      27.7        23.9
         F       5.9         7.0
men: 1198/2691 = 44.5%
women: 557/1835 = 30.4%
Departments where women's admission rate is higher: 4

The six-department subset contains 4,526 applications, so its totals must not be confused with the campus-wide figures covering 12,763 applications.

Within these six departments, women had higher admission rates in four departments and men in two.

Yet the combined rate was substantially higher for men.

This is not as perfectly symmetrical as our synthetic shipping example, where one option was better in every subgroup. It is nonetheless a powerful demonstration of how aggregation can obscure the pattern of subgroup relationships.

Why the department choices mattered

Look at departments A and B.

They admitted relatively high proportions of their applicants, and men accounted for most of the applications to those departments.

Now consider departments C, E, and F.

These departments were generally much more selective, and women accounted for a larger share of their applicants.

The combined admission rate therefore reflected two processes:

1. Admissions decisions within departments.

2. Differences in the departments to which applicants applied.

The original researchers found that the strong aggregate association favoring men was misleading when interpreted as evidence about the departmental admissions process. Their department-aware analysis reported a small statistically significant association favoring women after appropriate pooling.

That finding does not prove that every individual decision was fair, that applicants were equally situated in every respect, or that discrimination could not occur at other stages of education.

In fact, the researchers explicitly discussed how earlier social and educational processes could shape who entered different fields.

The broader lesson is that the level at which a decision is made matters to the level at which its outcomes should be analyzed.

A university-wide admission percentage may be useful for describing the university’s overall applicant outcomes. But when departments make admissions decisions independently, department-level comparisons are essential for evaluating their behavior.

This is what makes the Berkeley example so enduring: the aggregate statistics were real, the disparity was real, and yet their most obvious interpretation was not supported by the more detailed analysis.

10. Is Simpson’s paradox a mathematical error or a statistical error?

Neither, strictly speaking.

The underlying mathematics is correct.

The potential error occurs when we interpret the mathematics without understanding how the observations were generated.

Consider three different questions.

Question A: Which carrier actually delivered a larger proportion of its assigned shipments on time?

In our experiment, the answer is BudgetFlow.

Its observed aggregate rate was 86.5%, compared with 63.5% for ExpressSwift.

Question B: Which carrier had higher observed performance within each difficulty zone?

The answer is ExpressSwift.

It led by five percentage points in both zones.

Question C: Which carrier would perform better if we switched all shipments to it?

This is a causal question. The observed data alone does not guarantee an answer.

We might reasonably predict that ExpressSwift would perform better under a common shipment distribution if the within-zone rates remain stable.

But that assumption might fail.

For example, switching all shipments to ExpressSwift could overwhelm its logistics network, alter route assignments, increase delays, or change service quality.

The observational comparison does not tell us how those interventions would affect performance.

This distinction is important:

• Descriptive analysis measures what happened.

• Adjusted comparison attempts to make groups more comparable on measured characteristics.

• Causal inference attempts to estimate what would happen under an intervention.

Standardizing on delivery zone improves the comparison, but it does not necessarily eliminate every source of confounding.

Other relevant variables might include package weight, distance, weekday, weather, warehouse capacity, delivery deadline, or customer location.

If these variables differ systematically between carriers, the adjusted figures may still be misleading.

Similarly, adjusting for a variable that is itself affected by the carrier can create a different analytical problem. Not every available grouping variable should automatically be treated as a confounder.

The correct adjustment depends on the question and the process that generated the data.

11. How to detect Simpson’s paradox in real datasets

In everyday analytics, you rarely know in advance that a reversal exists.

You discover it by systematically checking aggregate patterns against comparable subgroups.

Step 1: Define the metric and its denominator

Before grouping anything, identify exactly what you are measuring.

For delivery reliability:

• Numerator: shipments delivered on time.

• Denominator: shipments eligible for the analysis.

• Rate: numerator divided by denominator.

For website conversion:

• Numerator: successful conversions.

• Denominator: eligible visitors or sessions, depending on the metric definition.

A change in the denominator can completely change the interpretation of a rate.

Make sure the numerator, denominator, time window, and eligibility rules remain consistent across groups.

Step 2: Identify variables that affect the outcome

Look for variables with strong relationships to both the outcome and the allocation of observations.

In shipping analytics, candidates include route difficulty, distance, and service level.

In e-commerce analytics, candidates include traffic source, customer segment, product category, and purchase intent.

In website experimentation, device type, geography, and traffic allocation may be relevant.

These variables provide useful starting points for subgroup analysis, but their suitability as adjustment variables requires domain knowledge.

Step 3: Inspect both subgroup rates and group sizes

A common mistake is reporting percentages without their corresponding counts.

Imagine seeing that a carrier has a 100% on-time rate in one group.

That could represent 10,000 successful deliveries out of 10,000, or just one successful delivery out of one.

The percentage looks identical, but the evidence is not equally informative.

Always review the number of observations supporting each rate.

Sparse groups can produce unstable estimates and apparent reversals driven partly by random variation.

Step 4: Compare the composition of the groups

A cross-tabulation of carrier by zone often exposes the reason for a reversal immediately.

Append this diagnostic to the delivery experiment:

zone_counts = (
    shipments.groupby(["carrier", "zone"])
    .size()
    .unstack(fill_value=0)
)

zone_shares = (
    zone_counts.div(
        zone_counts.sum(axis=1),
        axis=0,
    )
    .mul(100)
)

print("ZONE SHARES BY CARRIER (%)")
print(zone_shares.round(1).to_string())

The result:

ZONE SHARES BY CARRIER (%)
zone          Easy  Hard
carrier
BudgetFlow    90.0  10.0
ExpressSwift  10.0  90.0

This is the most revealing diagnostic in the experiment.

You can immediately see that one carrier operates primarily in easy environments while the other handles mostly difficult ones.

The aggregate comparison mixes operational performance with workload composition.

Step 5: Recalculate with meaningful shared weights

If a standard comparison is appropriate, choose a reference population and calculate adjusted rates.

Do not simply average subgroup percentages without examining their denominators.

An unweighted average gives every subgroup equal influence, which may or may not represent the population you care about.

In our example, an equal-zone average happens to be appropriate because the pooled data contains equal numbers of easy and hard deliveries.

That will not generally be true in a production dataset.

Step 6: Investigate uncertainty and missing groups

A reversal visible in a sample may not remain equally strong in future observations.

If a segment contains only a few shipments, its observed rate can be noisy.

For real analysis, consider confidence intervals, bootstrap resampling, or an appropriate statistical model.

Also inspect whether every option has enough observations in every important segment.

If one carrier never serves remote destinations, the data cannot directly estimate its performance there. Standardization would require extrapolation or additional evidence, not just a group-by operation.

A reliable analysis should make this lack of overlap visible rather than silently assuming that missing segment rates are known.

12. Common mistakes to avoid

Mistake 1: Treating aggregate performance as an intrinsic property

An overall metric depends on the population being measured.

A carrier’s success rate is influenced by both its ability to make deliveries and the difficulty of its assigned workload.

Likewise, a product’s conversion rate depends partly on which customers see it.

An aggregate rate does not automatically describe performance for an arbitrary future customer or shipment.

Mistake 2: Assuming every subgroup deserves equal weight

Equal weighting is a valid choice when the question calls for it.

But it is not automatically the statistically correct choice.

If 95% of future orders come from one customer segment, a business forecast may need to reflect that concentration.

The appropriate weight depends on the target population and intended decision.

Mistake 3: Assuming adjustment proves causation

Grouping by a relevant variable can remove some misleading associations, but it does not necessarily resolve selection effects or unmeasured confounding.

For a causal question, a randomized experiment may be more convincing when assignment is feasible and ethically appropriate.

Where randomization is unavailable, observational methods require explicit assumptions about confounding, selection, overlap, and the underlying causal structure.

Mistake 4: Ignoring changes in segment composition over time

Suppose the overall delivery rate improves from one month to the next.

Management might attribute that improvement to better logistics.

But perhaps the proportion of easy deliveries increased.

The operations team may have made no improvement within either zone.

This is a particularly important risk for dashboards.

A long-term trend in an aggregate metric can be driven by changing customer or workload composition rather than underlying performance changes.

Monitoring stable, relevant subgroups alongside overall metrics helps distinguish these possibilities.

Mistake 5: Assuming every reversal has the same explanation

Simpson’s paradox is an observable statistical pattern, not a single causal mechanism.

Reversals can arise from differences in subgroup sizes, sampling and selection processes, confounding, or other relationships among variables.

The grouping variable must be investigated in context.

You should not assume that any particular reversal proves discrimination, proves its absence, or automatically identifies the superior alternative.

13. A practical checklist for analysts

Before making a recommendation based on an aggregate metric, ask:

• Is the numerator and denominator defined consistently?
• Are the groups being compared exposed to similar conditions?
• Could another variable strongly influence the outcome?
• Have I examined performance within important subgroups?
• Have I checked the number of observations in each subgroup?
• Do the groups have substantially different subgroup distributions?
• Would standardization help answer the actual decision question?
• Have I chosen and documented an appropriate target population?
• Are the subgroup estimates sufficiently reliable?
• Am I making a descriptive claim or a causal claim?
• Would the conclusion change under a different reasonable weighting scheme?

You do not need to perform an elaborate causal analysis for every operational dashboard.

But when a metric will influence a consequential business decision, these checks can prevent a surprisingly common category of statistical mistake.

14. Final lessons: the answer depends on what you aggregate

Our shipping experiment started with an apparently convincing conclusion.

BudgetFlow delivered 86.5% of its shipments on time, while ExpressSwift managed only 63.5%.

Looking exclusively at those numbers, BudgetFlow appeared substantially better.

But the underlying shipment distributions were dramatically different.

Once we compared similar deliveries, the result reversed: ExpressSwift performed better in both easy and hard zones.

Using a common 50/50 distribution produced standardized rates of 77.5% for ExpressSwift and 72.5% for BudgetFlow.

The Berkeley admissions case showed why this phenomenon matters outside constructed examples. A pronounced aggregate association took on a very different interpretation once researchers examined the departments where admissions decisions were actually made.

The most important takeaways are:

• Aggregate statistics can be mathematically correct but inappropriate for the comparison you want to make.

• Different subgroup distributions can reverse the direction of an observed relationship.

• Grouped analysis reveals patterns that overall averages may conceal.

• Standardization makes comparisons use common weights, but those weights must match a meaningful target population.

• Neither a reversal nor an adjustment automatically establishes causality.

Above all, remember that an aggregate metric describes a mixture of outcomes and the composition of the population producing them.

When those two things are not separated, a seemingly obvious conclusion can point in exactly the opposite direction from the relevant subgroup evidence.

15. Your turn: reproduce the reversal on your own data

Run the Python experiment, verify the reversal assertions, and then change the numbers.

Try giving ExpressSwift more easy deliveries or making BudgetFlow’s hard-zone performance slightly better. Observe when the aggregate ranking flips and when it does not.

For an additional challenge, use an everyday dataset from your own projects—retail orders, website conversions, server latency, or delivery telemetry—and compare the overall metric with its important subgroups.

Then compute a standardized comparison using a target distribution relevant to your decision.

The goal is not to distrust every average. It is to understand what the average represents, which observations influence it, and whether it actually answers your question.

Your next step: Open your most important analytics dashboard, choose one headline percentage, and break it down by at least two meaningful segments. Check the segment sizes, compare the rates, and see whether the aggregate conclusion survives.

Because sometimes the most valuable statistical discovery is not a new trend. It is realizing that the trend you trusted was telling you the wrong story.