Hypothesis testing for engineers: p-values, effect sizes, and A/B pitfalls

1. Why hypothesis testing is an engineering problem

Imagine you’re responsible for the checkout experience of an e-commerce platform.

Your team has redesigned the checkout button. The new version has clearer wording, stronger visual contrast, and a more prominent position on mobile screens.

The product manager believes it will increase purchases. The designer is confident. The engineering team has already implemented the change behind a feature flag.

But should you release it to everyone?

You run an A/B test:

• Variant A (control): The existing checkout experience.

• Variant B (treatment): The redesigned checkout experience.

• Primary metric: The percentage of eligible users who complete a purchase.

• Business objective: Increase completed purchases without harming revenue per user or other important guardrail metrics.

After collecting data, you discover:

A 12% relative improvement sounds compelling. But is the difference convincing evidence of a real effect, or could it be random variation?

Even if the statistical evidence is strong, does the improvement justify deployment?

And what happens if your team checked the results every hour, stopping the experiment as soon as the p-value fell below 0.05?

These questions require more than a statistical library. They require understanding what an experiment can establish, what its assumptions are, and how engineering decisions can invalidate otherwise correct calculations.

By the end of this tutorial, you’ll be able to:

• Interpret p-values without treating them as probabilities that hypotheses are true.

• Calculate absolute effects, relative effects, and confidence intervals.

• Understand statistical power and minimum detectable effects.

• Reproduce false-positive inflation caused by repeatedly checking experiments.

• Implement a statistically valid sequential testing strategy using Python and SciPy.

• Detect common A/B testing failures, including sample ratio mismatches.

• Translate statistical evidence into defensible product decisions.

We’ll use synthetic e-commerce conversion data throughout, so every simulation is reproducible.

2. Hypothesis testing starts with two competing claims

A statistical hypothesis is a statement about an unknown property of the process generating your data.

For our checkout experiment, define:

• p_A = true probability that an eligible user purchases under control.

• p_B = true probability that an eligible user purchases under treatment.

The null hypothesis, H₀, says that the treatment does not change the purchase probability:

H₀: p_B − p_A = 0

The alternative hypothesis, H₁, says that the probabilities differ:

H₁: p_B − p_A ≠ 0

This is a two-sided test. We allow the treatment to help or hurt conversion.

A one-sided alternative would test specifically for an increase, but choosing that direction after observing the results is not acceptable. Direction, test statistic, and decision rule should be specified before analyzing outcomes.

Statistical evidence is not absolute proof

Classical hypothesis testing does not establish that a hypothesis is unquestionably true or false.

Instead, it asks whether the observed data are sufficiently incompatible with a particular statistical model.

Two mistakes are possible:

The significance level, alpha (α), controls the probability of a Type I error under a valid testing procedure and its assumptions.

A common choice is:

α = 0.05

For a single properly conducted test of a true null hypothesis, this means the probability of falsely rejecting H₀ is no greater than approximately 5%.

It does not mean that 5% of published significant findings are false. That depends on which hypotheses researchers test, statistical power, selection practices, and other factors.

It also does not mean an experiment has a 95% probability of being correct.

Statistical significance and practical significance differ

Suppose a large experiment estimates a conversion improvement from 10.00% to 10.05%.

With enough observations, that tiny difference could be statistically significant.

But whether it matters commercially depends on factors such as:

• The number of users exposed to the feature.

• The incremental profit generated per additional purchase.

• The engineering and infrastructure costs.

• Potential losses in customer experience or long-term retention.

• Opportunity costs relative to other improvements.

A statistically detectable improvement is not automatically a worthwhile improvement.

This distinction becomes clearer when we examine effect sizes.

3. Effect sizes: how much did the treatment actually change?

P-values address compatibility with a null hypothesis. Effect sizes describe the magnitude of an observed difference.

For binary conversion metrics, three quantities are especially useful.

Absolute difference

Absolute effect = conversion_B − conversion_A

In our example:

Absolute effect = 0.112 − 0.100 = 0.012

That is an improvement of 1.2 percentage points.

At the observed rates, the treatment produces 12 additional purchases per 1,000 eligible users.

This is often the most useful quantity for business forecasting.

Relative lift

Relative lift = (conversion_B − conversion_A) / conversion_A

For our checkout experiment:

Relative lift = 0.012 / 0.100 = 0.12 = 12%

Relative lift helps explain proportional improvements, but it can create misleading impressions when the baseline is small.

For example, improving conversion from 0.1% to 0.2% is a 100% relative improvement, yet only a 0.1-percentage-point absolute increase.

Always report both.

Risk ratio

For a binary outcome, the risk ratio is:

Risk ratio = conversion_B / conversion_A

Here:

Risk ratio = 0.112 / 0.100 = 1.12

The treatment’s estimated purchase probability is 1.12 times the control’s.

Despite the terminology, this measure applies to ordinary beneficial outcomes such as purchases and sign-ups.

An effect estimate is not necessarily the true effect

The observed 1.2-percentage-point improvement is a sample estimate.

With a different random sample, we might observe 0.6, 1.5, or even a negative difference.

That sampling uncertainty is why we need hypothesis tests and confidence intervals.

4. What a p-value actually means

A p-value measures how unusual an observed test statistic would be if the null hypothesis were true, under the assumptions of the statistical model.

More precisely:

p-value = probability, assuming H₀, of obtaining a test statistic at least as extreme as the observed statistic.

The definition matters because it is frequently misunderstood.

Consider our checkout experiment, where 1,000 out of 10,000 control users purchased, compared with 1,120 out of 10,000 treatment users.

For this binary outcome, we can use a two-proportion z-test.

The two-proportion z-test

The estimated conversion rates are:

p̂_A = purchases_A / users_A

p̂_B = purchases_B / users_B

Under the null hypothesis of equal conversion probabilities, the pooled estimate is:

p̂_pooled = (purchases_A + purchases_B) / (users_A + users_B)

The standard error under H₀ is:

SE₀ = √[p̂_pooled × (1 − p̂_pooled) × (1/n_A + 1/n_B)]

The test statistic is:

z = (p̂_B − p̂_A) / SE₀

For a two-sided test, the p-value is:

p-value = 2 × P(Z ≥ |z|)

Here, Z follows the standard normal distribution.

Using our observed data gives approximately:

• Pooled conversion rate: 10.6%.

• Observed absolute difference: 1.2 percentage points.

• z-statistic: 2.76.

• Two-sided p-value: 0.00584.

This is significant under a conventional fixed-horizon 5% test.

However, it is not significant under every possible experimental design, as we will demonstrate shortly.

Three incorrect interpretations

Incorrect: “There is a 0.584% probability that the null hypothesis is true.”

A frequentist p-value does not assign a probability to the hypothesis itself.

Incorrect: “There is a 99.416% probability the treatment works.”

The test does not directly calculate the probability that treatment is beneficial. That would require a different inferential framework, such as an explicitly specified Bayesian analysis.

Incorrect: “If p is below 0.05, the improvement must be meaningful.”

Statistical significance measures evidence against the null, not business value.

One additional complication is especially important for engineers: this p-value is calibrated for a specified testing procedure. Repeatedly analyzing accumulating data and stopping at a favorable result changes that procedure.

5. Confidence intervals: uncertainty belongs next to the estimate

A point estimate communicates magnitude. A confidence interval communicates uncertainty.

For our experiment, a large-sample approximation to the standard error of the difference is:

SE_effect = √[p̂_A(1 − p̂_A)/n_A + p̂_B(1 − p̂_B)/n_B]

For a conventional fixed-horizon 95% confidence interval:

CI = observed effect ± 1.96 × SE_effect

Using the checkout counts, this gives approximately:

FIXED-HORIZON EXAMPLE

Approximate 95% confidence interval: +0.35 to +2.05 percentage points

The teal interval represents the approximate fixed-sample confidence interval, with the dot marking the observed effect. It is not adjusted for optional stopping.A frequentist 95% confidence interval does not mean that there is a 95% probability the fixed, unknown effect lies in this particular observed interval.

It means that, across repetitions of the same valid procedure, approximately 95% of intervals constructed in this way would contain the true parameter.

This distinction is particularly important when results are selected after peeking or when multiple metrics are examined.

The normal-approximation interval above is appropriate for reasonably large independent Bernoulli samples. For very small samples, extreme conversion probabilities, or zero successes, prefer methods with better finite-sample behavior, such as score-based or exact intervals.

A further complication is that the confidence interval must be compatible with the experiment’s stopping and selection procedure.

We’ll address that in the sequential example.

6. Statistical power: a non-significant result does not prove no effect

Statistical power is the probability that a testing procedure rejects the null hypothesis under a specified alternative.

Power = 1 − β

Here, β is the probability of a Type II error for that particular alternative.

An experiment with 80% power to detect a 1.2-percentage-point absolute improvement has an 80% chance of detecting that effect, assuming the true effect is precisely that size and the model assumptions hold.

It does not have 80% power against every nonzero effect.

Minimum detectable effect

The minimum detectable effect (MDE) is generally the effect size chosen for sample-size planning to achieve a desired power at a specified significance level.

For our checkout experiment:

• Baseline conversion: 10%.

• Target absolute improvement: 1.2 percentage points.

• Significance level: 5%, two-sided.

• Target power: 80%.

• Allocation: 50/50 between variants.

A common rough normal-approximation calculation for equal-sized groups is:

n_per_arm ≈ 2 × p_avg × (1 − p_avg) × (z_α + z_power)² / δ²

Where:

• p_avg = average of the baseline and target conversion probabilities.

• δ = absolute target effect.

• z_α ≈ 1.96 for a two-sided 5% test.

• z_power ≈ 0.84 for 80% power.

This approximation suggests roughly 10,331 users per variant.

It assumes independent binary observations, no missing data, and a single fixed-horizon analysis.

A sequential experiment with stricter stopping boundaries can require more traffic to achieve similar power.

This is a useful engineering trade-off: frequent opportunities to stop early may come with more complex designs or a larger maximum sample size.

7. The experiment: reproduce p-hacking with Python and SciPy

Now we reach the core demonstration.

We will simulate thousands of checkout experiments and compare three strategies:

1. Fixed horizon: Collect the entire planned sample and test once at α = 0.05.

2. Naive peeking: Check the cumulative data ten times and declare success if any p-value falls below 0.05.

3. Planned sequential testing: Check ten times using a stricter, predeclared threshold that controls the probability of a false rejection across all looks.

The crucial experiment starts with a true null hypothesis.

Both variants will have exactly the same underlying conversion probability.

Any statistically significant result is therefore a false positive.

Step 1: Prepare the environment

Use Python 3.11 or newer.

Create a virtual environment and install NumPy and SciPy:

python -m venv .venv
source .venv/bin/activate
python -m pip install "numpy>=2.0,<3" "scipy>=1.16,<2"

On Windows PowerShell, activate the environment using its corresponding activation script.

The implementation uses established SciPy public APIs, including scipy.stats.norm and scipy.stats.binomtest. The latter provides exact binomial confidence intervals through proportion_ci. These interfaces are documented in the current SciPy reference.

SciPy v1.18.0 Manual+1

Step 2: Understand the simulation design

Each simulated experiment has:

• Ten planned interim analyses.

• An additional 1,000 users per variant between analyses.

• A maximum sample of 10,000 users per variant.

• Two independent Bernoulli conversion processes.

• A two-sided pooled two-proportion z-test at each look.

We’ll simulate 20,000 independent experiments under the null and another 20,000 with a genuine improvement.

Importantly, the repeated observations in the interim analyses are cumulative. The second analysis includes the first batch, the third includes the first two, and so on.

This reproduces how real experiment dashboards work.

Step 3: Run the complete simulation

Save the following as ab_test.py.

The script requires no downloaded dataset and generates its own synthetic observations.

import numpy as np
from scipy.stats import binomtest, norm

ALPHA = 0.05
LOOKS = 10
BATCH_PER_ARM = 1_000
SIMULATIONS = 20_000

def two_sided_pvalues(success_a, success_b, n_per_arm):
    """Pooled two-proportion z-test for cumulative counts."""

    rate_a = success_a / n_per_arm
    rate_b = success_b / n_per_arm

    pooled = (success_a + success_b) / (2 * n_per_arm)

    standard_error = np.sqrt(
        2 * pooled * (1 - pooled) / n_per_arm
    )

    z = np.divide(
        rate_b - rate_a,
        standard_error,
        out=np.zeros_like(standard_error, dtype=float),
        where=standard_error > 0,
    )

    return 2 * norm.sf(np.abs(z))

def simulate(p_a, p_b, seed):
    """Compare fixed, naive-peeking, and sequential policies."""

    rng = np.random.default_rng(seed)
    shape = (SIMULATIONS, LOOKS)

    cumulative_a = rng.binomial(
        BATCH_PER_ARM, p_a, size=shape
    ).cumsum(axis=1)

    cumulative_b = rng.binomial(
        BATCH_PER_ARM, p_b, size=shape
    ).cumsum(axis=1)

    n_per_arm = BATCH_PER_ARM * np.arange(
        1, LOOKS + 1
    )

    pvalues = two_sided_pvalues(
        cumulative_a, cumulative_b, n_per_arm
    )

    fixed_horizon = pvalues[:, -1] < ALPHA

    naive_peeking = (
        pvalues < ALPHA
    ).any(axis=1)

    planned_sequential = (
        pvalues < ALPHA / LOOKS
    ).any(axis=1)

    return {
        "fixed_horizon": fixed_horizon.mean(),
        "naive_peeking": naive_peeking.mean(),
        "planned_sequential": planned_sequential.mean(),
    }

def simultaneous_difference_interval(
    success_a, success_b, n_per_arm
):
    """Conservative 95% interval across ten planned looks."""

    confidence = 1 - ALPHA / (2 * LOOKS)

    ci_a = binomtest(
        success_a, n_per_arm
    ).proportion_ci(
        confidence_level=confidence,
        method="exact",
    )

    ci_b = binomtest(
        success_b, n_per_arm
    ).proportion_ci(
        confidence_level=confidence,
        method="exact",
    )

    return (
        ci_b.low - ci_a.high,
        ci_b.high - ci_a.low,
    )

for label, rates, seed in [
    ("Null: same conversion", (0.10, 0.10), 2026),
    ("Real +1.2 pp effect", (0.10, 0.112), 2027),
]:
    outcomes = simulate(*rates, seed=seed)

    print(f"\n{label}")

    for name, fraction in outcomes.items():
        print(f"  {name:>20}: {fraction:.2%}")

success_a, success_b, n = 1_000, 1_120, 10_000

observed_effect = (success_b - success_a) / n

sample_p = two_sided_pvalues(
    np.array([success_a]),
    np.array([success_b]),
    n,
)[0]

fixed_se = np.sqrt(
    (success_a / n) * (1 - success_a / n) / n
    + (success_b / n) * (1 - success_b / n) / n
)

critical = norm.isf(ALPHA / 2)

fixed_ci = (
    observed_effect - critical * fixed_se,
    observed_effect + critical * fixed_se,
)

seq_ci = simultaneous_difference_interval(
    success_a, success_b, n
)

print("\nWorked example: 1000/10000 versus 1120/10000")

print(f"  observed absolute lift: {observed_effect:.2%}")
print(f"  unadjusted p-value: {sample_p:.5f}")

print(
    "  fixed-horizon approximate 95% CI: "
    f"({fixed_ci[0]:.2%}, {fixed_ci[1]:.2%})"
)

print(
    "  sequential-safe conservative 95% CI: "
    f"({seq_ci[0]:.2%}, {seq_ci[1]:.2%})"
)

print(f"  fixed-horizon reject: {sample_p < ALPHA}")

print(
    "  10-look sequential threshold at this look: "
    f"{sample_p < ALPHA / LOOKS}"
)

n_a, n_b = 9_700, 10_300

srm_p = binomtest(
    n_b, n_a + n_b, p=0.5
).pvalue

print(
    "\nSample ratio mismatch diagnostic p-value: "
    f"{srm_p:.7f}"
)

baseline, delta = 0.10, 0.012
average_rate = baseline + delta / 2

z_alpha = norm.isf(ALPHA / 2)
z_power = norm.isf(0.20)

rough_n = (
    2
    * average_rate
    * (1 - average_rate)
    * (z_alpha + z_power) ** 2
    / delta**2
)

print(
    "Rough per-arm n for 80% power (fixed horizon): "
    f"{np.ceil(rough_n):.0f}"
)

Run it from your terminal:

python ab_test.py

The random number generator has explicit seeds, so results are reproducible with the same NumPy random generation behavior.

The large-sample z-test used here is suitable for the simulated conversion probabilities and group sizes. It should not be generalized blindly to extremely sparse outcomes or correlated observations.

8. Results: repeated peeking dramatically increases false positives

Here are the results from the seeded simulation, rounded to two decimal places.

Null: same conversion
         fixed_horizon: 4.77%
         naive_peeking: 19.68%
    planned_sequential: 2.46%

Real +1.2 pp effect
         fixed_horizon: 78.58%
         naive_peeking: 85.92%
    planned_sequential: 56.42%

Worked example: 1000/10000 versus 1120/10000
  observed absolute lift: 1.20%
  unadjusted p-value: 0.00584
  fixed-horizon approximate 95% CI: (0.35%, 2.05%)
  sequential-safe conservative 95% CI: (-0.67%, 3.07%)
  fixed-horizon reject: True
  10-look sequential threshold at this look: False

Sample ratio mismatch diagnostic p-value: 0.0000228
Rough per-arm n for 80% power (fixed horizon): 10331

The false-positive experiment

False-positive rate when there is no real effect20,000 simulated experiments, ten planned looks; lower is better

Simulated probabilities; Bonferroni correction is deliberately conservative.

The fixed-horizon test behaves as expected: approximately 5% of experiments incorrectly reject the null.

The naive peeking strategy behaves very differently.

It produces false discoveries in 19.68% of experiments, despite the treatment having absolutely no effect.

That is nearly four times the intended false-positive rate.

The reason is straightforward. We gave the experiment multiple opportunities to appear significant and accepted the first favorable result.

The reported p-value still looks like a conventional p-value, but the overall decision procedure no longer has a 5% false-positive rate.

Why repeated testing creates this problem

Imagine monitoring a dashboard every morning.

On Monday, the treatment appears worse. On Tuesday, the difference is almost zero. On Wednesday, the treatment appears better.

Eventually, random fluctuations may produce a p-value below 0.05.

If you stop immediately and celebrate, you have selected an unusually favorable observation.

This is a form of optional stopping, often called peeking when applied to repeated interim checks.

The important distinction is that looking at accumulating data is not inherently invalid. What matters is how the data influence stopping, selection, and final decisions.

A team can inspect operational dashboards while preserving a precommitted fixed-horizon analysis, provided those inspections do not cause unplanned changes to the statistical decision procedure.

The dangerous practice is changing the stopping rule based on interim evidence while continuing to interpret ordinary p-values as though only one test had occurred.

9. The correct sequential approach: control error across all planned looks

We want the operational flexibility of interim analyses without sacrificing statistical validity.

A sequential design allows us to make decisions as data accumulate, using decision boundaries selected to account for repeated testing.

There are several legitimate approaches:

• Group-sequential methods with calibrated stopping boundaries.

• Alpha-spending designs.

• Always-valid p-values and appropriate sequential tests.

• E-values or nonnegative test martingales under suitable assumptions.

For this tutorial, we’ll use a particularly transparent approach: Bonferroni correction across ten planned analyses.

It is conservative but easy to implement, audit, and verify.

Step 1: Predeclare the analyses

Before the experiment begins, specify:

• Maximum of ten interim looks.

• First look at 1,000 users per variant.

• Subsequent looks every additional 1,000 users per variant.

• Maximum of 10,000 users per variant.

• Overall desired Type I error limit of 5%.

• Rejection if the two-sided p-value is below 0.005 at any planned look.

The per-look threshold is:

α_look = α_total / number_of_looks

α_look = 0.05 / 10 = 0.005

Step 2: Understand why the correction works

Suppose E₁, E₂, …, E₁₀ represent false rejections at the ten analyses.

The probability of at least one false rejection obeys the union bound:

P(any false rejection) ≤ P(E₁) + P(E₂) + … + P(E₁₀)

If each valid marginal test rejects a true null with probability no greater than 0.005:

P(any false rejection) ≤ 10 × 0.005 = 0.05

This result does not require the ten analyses to be statistically independent.

That is particularly useful because cumulative A/B testing results are strongly correlated.

The Bonferroni rule also remains valid if the experiment stops after the first crossing, as long as its planned looks and inferential assumptions are respected.

In our simulation, the corrected strategy produces a false-positive rate of approximately 2.46%.

That is below the required 5% limit.

Why the result is below 5%

Bonferroni controls an upper bound rather than making the actual false-positive probability equal to that bound.

Because cumulative test statistics are correlated, the chances of false positives at different looks overlap.

Consequently, the total probability of a false rejection is smaller than the sum of the individual probabilities.

This protection has a cost: the corrected procedure is conservative.

The power trade-off

Probability of detecting a genuine improvementControl conversion 10%, treatment conversion 11.2%; maximum 10,000 users per variant

Naive peeking has inflated false positives, so its apparent detection advantage is not a fair operating-point comparison.

Under a real 1.2-percentage-point improvement:

• The fixed-horizon procedure detects the improvement in approximately 79% of simulations.

• Naive peeking appears more powerful, but its Type I error is uncontrolled at the intended 5% level.

• The corrected sequential procedure detects the improvement in approximately 56% of simulations.

The sequential method is statistically valid, but less powerful at the same maximum sample size.

If early detection and 80% power are both important, a better-designed group-sequential procedure may be preferable.

For example, O’Brien–Fleming-style boundaries require particularly strong evidence early and become less stringent later. Properly calibrated group-sequential methods can make more efficient use of an overall error budget than our simple equal-threshold rule.

However, those boundaries must be calculated for the actual analysis schedule and statistical model. They cannot safely be replaced with arbitrary changing thresholds.

What engineers should remember

Sequential testing is not simply “run the normal test more often.”

It requires designing the whole sequence of decisions.

That means defining the possible analyses, error allocation, stopping criteria, and interpretation of results before observing the experiment outcomes.

10. Confidence intervals under sequential stopping

There is another subtle issue.

Suppose your corrected sequential experiment stops early because it crosses its significance threshold.

Can you calculate a conventional 95% confidence interval using the stopped sample and interpret it exactly as though the sample size were fixed beforehand?

Not generally.

A fixed-horizon confidence interval may lose its stated coverage guarantees when applied at a data-dependent stopping time.

This is a common reason why apparently sophisticated A/B testing pipelines still produce misleading estimates.

A conservative solution using exact binomial intervals

Our code implements a simple simultaneous confidence interval that remains valid at any of the ten prespecified looks.

For each variant and each look, construct an exact binomial confidence interval.

There are:

2 variants × 10 looks = 20 individual intervals.

Give each individual interval a noncoverage allowance of:

0.05 / 20 = 0.0025

This corresponds to an individual confidence level of 99.75%.

By the union bound, all 20 intervals simultaneously contain their respective true conversion probabilities with probability at least 95%.

At any planned stopping look, we can then construct an interval for the treatment difference:

Lower difference bound = lower_B − upper_A

Upper difference bound = upper_B − lower_A

Because all component intervals are simultaneously valid with the required probability, the selected difference interval inherits at least 95% coverage.

SciPy’s exact binomial interval makes this construction straightforward.

SciPy v1.18.0 Manual

Interpreting the worked example

For our observed counts, we found:

The fixed-horizon interval excludes zero.

The sequential-safe interval includes zero.

These are not contradictory statements. They correspond to different inferential procedures and different coverage guarantees.

The sequential interval is deliberately conservative. Its width reflects both exact binomial uncertainty and protection across multiple possible stopping times.

It is also not specifically calibrated to invert the p-value rejection rule used earlier, so its inclusion of zero need not agree exactly with that rule’s significance decision.

For a production experimentation platform, choose compatible sequential hypothesis tests and interval estimators if you require such agreement.

For exploratory engineering work, this construction provides a useful, understandable safeguard against overly confident conclusions.

11. P-hacking extends far beyond peeking

Repeatedly testing accumulating data is only one way to manufacture misleading significance.

Other practices can have similar effects.

Trying many metrics

Suppose a team tests twenty independent metrics, each under a true null hypothesis, using a 5% threshold.

The probability that at least one metric appears significant is:

P(at least one false positive) = 1 − (1 − 0.05)²⁰

That is approximately 64%.

Real product metrics are usually correlated, so this independent-tests calculation is an illustration rather than a universal estimate.

Still, it demonstrates why reporting only the most favorable metric is dangerous.

A better approach is to define a primary metric, identify secondary and exploratory metrics, and apply an appropriate multiplicity strategy to claims requiring error control.

Searching for winning segments

Imagine the checkout redesign performs poorly overall.

The team then inspects results across:

• Desktop and mobile users.

• New and returning customers.

• Ten countries.

• Several browser types.

• Multiple acquisition channels.

Eventually, one segment shows a substantial improvement.

That may reflect genuine treatment-effect heterogeneity. It may also simply be a chance finding produced by many comparisons.

The correct response is not to ignore potentially valuable segments. Instead:

1. Treat newly discovered segments as exploratory.

2. Estimate effect sizes and uncertainty.

3. Account for the number and dependence of comparisons when making formal claims.

4. Validate promising findings in an independent experiment or using a prespecified confirmatory analysis.

Finding a segment where one comparison is significant and another is not does not itself establish that the treatment effects differ between those segments.

That claim requires testing the difference between effects, often through an interaction analysis.

Changing the primary metric after seeing results

Suppose conversion decreases but click-through rate increases.

If the team originally defined conversion as the primary metric, promoting click-through rate after observing the outcome changes the question being answered.

The click-through improvement may be interesting, but it should not retrospectively replace the prespecified success criterion without acknowledging the exploratory nature of the change.

Removing inconvenient data

It is legitimate to remove corrupted records using defensible quality rules.

It is not legitimate to keep adjusting exclusion criteria until the experiment becomes significant.

Good experimentation systems version their eligibility logic, record exclusions, and preserve enough audit information to reconstruct the analysis.

12. Sample ratio mismatch: when your randomization pipeline lies

Before interpreting effect sizes or p-values, verify that the experiment’s observed population is consistent with the intended allocation.

A sample ratio mismatch, or SRM, occurs when the observed allocation differs statistically from the expected allocation.

Suppose an experiment was configured for a 50/50 split.

After processing the data, the analysis contains:

• 9,700 users assigned to control.

• 10,300 users assigned to treatment.

Unequal counts do not automatically prove a bug. Random allocation naturally produces some imbalance.

The question is whether the discrepancy is plausible under the configured randomization process.

Our code calculates a two-sided exact binomial p-value for this example:

Sample ratio mismatch diagnostic p-value: 0.0000228

Such a small p-value is a strong reason to investigate the allocation or measurement pipeline.

Possible causes include:

• Incorrect feature-flag bucketing.

• Missing exposure events in one variant.

• Analytics filters that behave differently between variants.

• Users switching treatment assignments.

• Duplicate user identities.

• Inconsistent event joins.

• Treatment-dependent event collection.

The check should use the appropriate unique experimental units and expected allocation probabilities. If users are deliberately allocated 70/30, testing against 50/50 would be incorrect.

A mismatch test is a diagnostic, not proof of a particular bug. It also does not guarantee that a balanced experiment is unbiased.

13. The cherry on the cake: Microsoft’s experiment that reversed its conclusion

One of the most instructive real-world A/B testing failures was documented by Microsoft’s experimentation team.

The case involved an MSN image carousel experiment.

The change: Increase the number of rotating carousel cards from 12 to 16.

The surprising result: Initial analysis suggested that the redesigned carousel reduced user engagement, contrary to expectations.

The warning: The experiment exhibited a sample ratio mismatch.

The team investigated rather than accepting the initial result.

According to Microsoft’s published account, users exposed to the more engaging carousel interacted enough to trigger a bot detection algorithm. That algorithm disproportionately filtered engaged users out of the treatment data.

In other words, the treatment affected which users remained visible to the analysis.

After correcting the issue, the result reversed: the redesigned carousel actually increased engagement.

Microsoft described the incident in its September 2020 article, Diagnosing Sample Ratio Mismatch in A/B Testing.

Microsoft Research

This is a documented historical incident, not a claim about a newly discovered 2026 bug.

Why this case matters

The problem was not merely insufficient sample size or a poorly chosen statistical test.

The underlying observation process was biased.

A flawless significance calculation applied to biased data can still produce the wrong product decision.

The most important lesson is that a trustworthy experiment requires more than correct statistics:

• Assignment must be reliable.

• Events must be collected consistently.

• Inclusion and exclusion rules must not introduce treatment-dependent selection bias.

• Metrics must correspond to actual product behavior.

• Data quality checks must be completed before drawing causal conclusions.

A/B testing is therefore as much a distributed-systems and data-engineering problem as a statistics problem.

14. From statistical results to deployment decisions

A production decision should combine evidence, effect size, uncertainty, and engineering risk.

Consider four possible outcomes.

The last row deserves attention.

Failing to reject a null hypothesis does not establish equivalence.

If the business question is whether two implementations are sufficiently similar, an equivalence test with a prespecified practical margin may be more appropriate than an ordinary difference test.

If the question is whether treatment is no worse than control by more than an acceptable amount, consider a non-inferiority design.

The test should match the decision, not the other way around.

Make the experimental unit explicit

Our simulations treat users as independent Bernoulli observations, with one purchase indicator per user.

That is a simplified but coherent data-generating process.

In real applications, one user might:

• Browse multiple times.

• Use several devices.

• Generate many click events.

• Purchase more than once.

• Interact with other users.

If assignment occurs at the user level, treating every click as an independent observation can underestimate uncertainty.

The experimental unit, randomization unit, and statistical analysis must be compatible.

For example, when users are assigned to variants, a per-user binary purchase metric can be modeled with independent observations when the independence assumption across users is reasonable.

For metrics such as average revenue per user, user-level aggregation or appropriately clustered inference is typically more defensible than pretending every transaction is independent.

Interference between users, shared accounts, marketplace effects, and network interactions can require more specialized experimental designs.

Watch for delayed outcomes

Many product metrics are not immediately observable.

Suppose success is defined as purchasing within seven days of first exposure.

A user exposed yesterday has not yet completed the full observation window.

Comparing that user’s incomplete conversion history with the completed history of earlier users introduces potential bias.

You need an analysis plan specifying:

• The triggering or eligibility event.

• When the outcome window begins.

• How long outcomes are observed.

• How incomplete observation windows are handled.

• When the dataset is considered mature enough for a planned analysis.

Our simulation assumes immediately observed binary outcomes, so it does not model these delays.

A production implementation must.

15. An engineering checklist for trustworthy A/B tests

Before launching an experiment, review the following.

Pre-launch and analysis checklist

0 of 10 completed

Define one primary decision metric and its eligibility rulesSpecify the randomization unit and keep assignments stableChoose the minimum effect worth detecting and calculate powerSelect a fixed-horizon or valid sequential analysis planPredeclare stopping rules, looks, and significance thresholdsInstrument exposure and outcome events consistentlyCheck the expected allocation and investigate sample ratio mismatchesReport absolute effects, relative effects, and appropriate intervalsSeparate exploratory findings from confirmatory conclusionsCheck guardrail metrics and delayed outcomes before shippingAlso record the experiment configuration in a version-controlled document or experiment registry.

The purpose is not bureaucracy. It is to make the decision process reconstructable when the results are surprising, disputed, or reused months later.

16. What to improve next in the simulation

The example is intentionally small enough to audit, but there are several valuable extensions.

First, compare sequential boundaries. Implement a properly calibrated group-sequential design and compare its Type I error, power, average sample size, and maximum sample size against the Bonferroni method.

Second, investigate different effect sizes. Re-run the simulator with absolute improvements of 0, 0.2, 0.5, and 1.2 percentage points. Plot detection probability against the true effect.

Third, simulate heterogeneous users. Generate different baseline purchase probabilities for new and returning users. Randomize treatment within strata and compare prespecified adjusted analysis with an unadjusted estimate.

Fourth, simulate broken instrumentation. Make treatment users with successful purchases more likely to disappear from the analysis. Observe whether the experiment can produce an incorrect treatment conclusion despite generating a real positive effect.

Fifth, measure the stopping time. For each experiment that crosses a sequential boundary, record the first crossing and the corresponding sample size. Compare the mean sample size to the fixed-horizon requirement, including experiments that reach the maximum sample without rejecting.

The last exercise is especially valuable because legitimate sequential designs offer benefits beyond discovering significant results more quickly. They may reduce exposure to harmful changes or stop for futility, provided those objectives and decisions are incorporated correctly into the design.

17. Final takeaways

Hypothesis testing is most useful when treated as a complete decision procedure rather than a calculation performed at the end of data collection.

The central principles are:

• P-values are conditional evidence, not probabilities that hypotheses are true.

• Effect sizes communicate magnitude; confidence intervals communicate uncertainty.

• Statistical significance is not synonymous with business value.

• Repeatedly testing accumulating data with an unadjusted threshold can dramatically inflate false positives.

• Sequential testing is valid when stopping rules and inferential procedures account for repeated analyses.

• Reliable randomization, instrumentation, and data quality are prerequisites for reliable statistical conclusions.

Our simulation demonstrated the central failure clearly.

With no underlying treatment effect, a fixed-horizon procedure produced approximately 5% false positives, while naive peeking produced about 20%.

A preplanned sequential correction restored error control, although at the cost of lower power for the same maximum sample size.

That trade-off is not a reason to avoid sequential testing. It is a reason to design it deliberately.

Your next step: make the experiment harder

Run the supplied simulation, then change the number of interim analyses from 10 to 50 while preserving the same maximum sample size.

Predict what will happen to naive peeking before executing the modified experiment.

Then compare the results with a correctly adjusted sequential procedure.

Finally, introduce a small genuine conversion improvement and measure how the two valid testing strategies trade statistical power for opportunities to stop early.

The goal is not to produce a p-value below 0.05. The goal is to build an experiment whose conclusions remain trustworthy even when the outcome is surprising.

That is the standard engineers should bring to every production A/B test.