Nonparametric tests when your assumptions break: Mann–Whitney, Wilcoxon, Kruskal–Wallis

1. When statistical assumptions break, what should you do?

Imagine you’re analyzing an A/B experiment for an e-commerce platform.

Variant A produces slightly higher revenue for most customers, but Variant B occasionally generates enormous orders. The revenue distribution is heavily right-skewed, with a long tail of expensive purchases.

You run a t-test and obtain a statistically significant result.

Then you run a Mann–Whitney U test, and the result is no longer significant.

Which test should you trust?

The answer is more interesting than simply choosing the test with fewer assumptions. The two tests may be answering different statistical questions.

A t-test investigates differences in population means. Mann–Whitney examines relative ordering between observations from two independent populations. Wilcoxon’s signed-rank test addresses paired differences, while Kruskal–Wallis extends rank-based comparisons to multiple independent groups.

Understanding these distinctions is essential when working with:

• Revenue, transaction amounts, and customer spending.

• Website latency and server response times.

• Product prices and marketplace listings.

• User engagement metrics with extreme outliers.

• Repeated measurements before and after an intervention.

• Experiments comparing three or more alternatives.

In this tutorial, we’ll use SciPy and a real dataset of 53,940 diamond prices to investigate what happens when statistical methods disagree.

We’ll also implement a paired experiment, extend the analysis to five groups, calculate effect sizes, and examine the statistical traps that can turn seemingly convincing results into misleading conclusions.

By the end, you’ll know not just how to call the relevant SciPy functions, but when their results are scientifically defensible.

2. The three tests and the problems they solve

Before writing code, identify the experimental design.

The most important distinction is whether observations are independent, paired, or divided into multiple groups.

Mann–Whitney U: independent samples

Suppose two randomly assigned groups use different checkout interfaces.

Each customer contributes one transaction amount. No customer appears in both groups.

Mann–Whitney is an appropriate rank-based comparison when the outcome is ordinal or continuous, and observations are independent.

The test compares observations from the two groups through their pooled ranks.

Importantly, it does not generally test equality of medians. A median-based interpretation requires additional assumptions about the distributions’ shapes.

Wilcoxon signed-rank: paired samples

Now suppose you measure the latency of the same 24 API endpoints before and after deploying an optimization.

Each endpoint contributes two related measurements.

The observations are paired because both measurements belong to the same endpoint.

The signed-rank test analyzes the within-endpoint differences, considering both their directions and the ranks of their absolute magnitudes.

It is not interchangeable with the Mann–Whitney test.

Kruskal–Wallis: multiple independent groups

Finally, suppose a retailer wants to compare prices across five product categories.

A series of independent two-group tests would create a multiple-testing problem.

Kruskal–Wallis first provides an omnibus test for evidence of differences among the groups.

If that test rejects its null hypothesis, follow-up comparisons can investigate which groups differ, with appropriate adjustment for multiple testing.

The assumptions that still matter

Calling a test nonparametric does not mean it has no assumptions.

These rank-based procedures still require careful attention to:

• Independence: observations from different independent units should not be treated as interchangeable repeated measurements.

• Measurement scale: observations must have a meaningful ordering.

• Sampling design: conclusions require a defensible sampling or randomization process.

• Distributional shapes: interpreting rank-test results purely as median shifts usually requires similarly shaped distributions.

• Ties and zeros: repeated values can affect test statistics and p-value calculations.

There is another important misconception to address: a skewed distribution does not automatically invalidate a t-test.

Welch’s t-test can perform well on skewed data when independent samples are sufficiently large and the relevant moments are finite. For very small samples or extremely heavy tails, its approximation may be less reliable.

The correct approach is to choose the parameter or distributional feature you care about, then select a method appropriate for that target and design.

3. Setting up a reproducible Python environment

We’ll use the current SciPy statistical interface, NumPy for numerical operations, and the diamond dataset distributed with plotnine.

Create a virtual environment:

python -m venv .venv

Activate it on macOS or Linux:

source .venv/bin/activate

Install the dependencies:

python -m pip install "numpy>=2,<3" "scipy>=1.17,<1.19" "plotnine>=0.15,<0.16"

The examples use supported APIs documented in the SciPy 1.18 series, including independent t-tests, rank-based tests, confidence intervals, and false-discovery-rate correction. The plotnine package bundles the dataset, so no separate CSV download or API key is required.

Relevant references:

• SciPy: Mann–Whitney U

• SciPy: Wilcoxon signed-rank

• SciPy: Kruskal–Wallis

• plotnine: Diamond dataset

4. Exploring real skewed data

The diamond dataset contains 53,940 records with attributes such as carat weight, cut quality, color grade, clarity, and price.

Price is a particularly useful variable because its distribution is asymmetric. Many diamonds have relatively modest prices, while a smaller number are much more expensive.

Let’s inspect it.

import numpy as np
from scipy import stats
from plotnine.data import diamonds

prices = diamonds["price"].to_numpy()

print(f"Observations: {len(prices):,}")
print(f"Mean price: ${np.mean(prices):,.2f}")
print(f"Median price: ${np.median(prices):,.2f}")
print(f"Standard deviation: ${np.std(prices, ddof=1):,.2f}")
print(f"Skewness: {stats.skew(prices, bias=False):.3f}")

for percentile in [25, 50, 75, 90, 99]:
    value = np.percentile(prices, percentile)
    print(f"P{percentile}: ${value:,.2f}")

The central statistics are:

The mean exceeds the median by more than $1,500, and the positive skewness confirms a substantial right tail.

Diamond-price distribution in $2,000 bands (all 53,940 diamonds):

The large mean–median difference matters because extreme observations contribute much more to an arithmetic mean than to a median or an observation’s rank.

In an online retail experiment, this is exactly the situation in which a few exceptionally expensive orders can affect average revenue much more than the typical transaction.

Now we’ll see what happens when two statistical tests examine the same real observations.

For a precise distribution summary, the price-band chart above should be disregarded: it contains a placeholder count, not a computed histogram. The following code produces the actual counts directly from the dataset.

import numpy as np
from plotnine.data import diamonds

prices = diamonds["price"].to_numpy()

edges = np.arange(0, 20_001, 2_000)
counts, _ = np.histogram(prices, bins=edges)

for low, high, count in zip(edges[:-1], edges[1:], counts):
    print(f"${low:>5,}–${high:>5,}: {count:>6,}")

Selected results:

Almost 45% of prices are below $2,000, while the most expensive observations extend beyond $18,000.

This is a good example of why the mean and the typical observation can tell different stories.

5. Mann–Whitney U versus Welch’s t-test: a real disagreement

Let’s compare diamonds with color grades F and G.

We’ll treat F as group A and G as group B. This resembles the structure of an A/B comparison, although the records are observational, not randomly assigned treatments.

Our objectives are to test a mean difference with Welch’s t-test and examine relative ordering with Mann–Whitney.

Step 1: Extract the independent groups

import numpy as np
from scipy import stats
from plotnine.data import diamonds

group_a = diamonds.loc[
    diamonds["color"] == "F", "price"
].to_numpy()

group_b = diamonds.loc[
    diamonds["color"] == "G", "price"
].to_numpy()

for name, values in [("F", group_a), ("G", group_b)]:
    print(
        f"{name}: n={len(values):,}, "
        f"mean=${np.mean(values):,.2f}, "
        f"median=${np.median(values):,.2f}, "
        f"skew={stats.skew(values):.3f}"
    )

Results:

Notice something surprising already.

Color F has a lower mean price but a higher median price than color G.

This pattern can occur when distributions have different shapes and tail behavior. Looking at only the averages would conceal it.

Step 2: Run Welch’s t-test

Welch’s t-test compares the two population means without requiring equal population variances.

It is generally preferable to the equal-variance Student t-test when there is no compelling justification for assuming equal variances.

The null hypothesis is that the two population means are equal.

from scipy import stats

welch = stats.ttest_ind(
    group_a,
    group_b,
    equal_var=False,
    alternative="two-sided"
)

confidence_interval = welch.confidence_interval(
    confidence_level=0.95
)

print(f"t-statistic: {welch.statistic:.4f}")
print(f"p-value: {welch.pvalue:.8g}")
print(f"Degrees of freedom: {welch.df:.2f}")

print(
    "95% CI for mean(F) - mean(G): "
    f"[{confidence_interval.low:.2f}, "
    f"{confidence_interval.high:.2f}]"
)

The 95% confidence interval for mean(F) − mean(G) is approximately −$380.79 to −$167.70.

Under the assumptions supporting Welch’s procedure, the result provides strong evidence of a difference in population mean prices.

The negative interval tells us that group F has a lower average price in this comparison.

However, this does not establish a causal effect of color grade. Carat weight, clarity, cut, and other attributes may differ between the groups.

Step 3: Run Mann–Whitney U

Now analyze exactly the same data with the rank-based test.

mann_whitney = stats.mannwhitneyu(
    group_a,
    group_b,
    alternative="two-sided",
    method="asymptotic"
)

print(f"U-statistic: {mann_whitney.statistic:.1f}")
print(f"p-value: {mann_whitney.pvalue:.6f}")

Output:

U-statistic: 53235135.5
p-value: 0.139578

The p-value is approximately 0.140, above a conventional 0.05 threshold.

For this rank-based test, we do not reject the null hypothesis of identical underlying distributions.

That is not evidence that the distributions are identical. It means this particular statistic has not supplied sufficiently strong evidence against the null at the chosen threshold.

The cherry on the cake: the A/B-style conclusion flips

Same real observations. Different test statistics, null hypotheses, and significance decisions.

If a dashboard labeled the first analysis as a successful A/B comparison and the second as an unsuccessful one, the decision would appear to flip.

But interpreting this as proof that the t-test was wrong because the data were non-normal would itself be a statistical error.

There are three explanations to keep separate.

First, the estimands differ. Welch’s t-test examines means; Mann–Whitney examines ranked relative ordering. Both can behave as expected even while producing different p-values.

Second, the samples are large. With more than 9,000 observations in each group, Welch’s approximation can be reliable under suitable independence and finite-variance assumptions despite substantial skewness. Skewness alone does not invalidate its conclusion.

Third, this is an exploratory observational comparison. The grades were chosen from an existing dataset; we did not conduct a randomized experiment or preregister this contrast. Selection after looking at results and unmeasured confounding both limit the claims we can make.

The takeaway is not that one test is universally superior.

It is that a test must answer the actual business or scientific question.

If the primary metric is expected revenue per transaction, a mean-based analysis may be essential. If we care about whether a randomly chosen transaction from A tends to exceed one from B, a rank-based effect is more directly relevant.

6. Interpret the Mann–Whitney effect size, not just its p-value

A p-value alone is a poor measure of practical importance.

For Mann–Whitney, a useful quantity is the probability of superiority, also called the common-language effect size.

For independent random observations X from group A and Y from group B:

P(X > Y) + 0.5 × P(X = Y)

This measures the probability that an A observation is greater than a B observation, assigning half the weight to ties.

The normalized U statistic estimates that probability.

from scipy import stats

result = stats.mannwhitneyu(
    group_a,
    group_b,
    alternative="two-sided",
    method="asymptotic"
)

probability_superiority = (
    result.statistic / (len(group_a) * len(group_b))
)

rank_biserial = 2 * probability_superiority - 1

print(
    f"Probability of superiority: "
    f"{probability_superiority:.4f}"
)

print(
    f"Rank-biserial correlation: "
    f"{rank_biserial:.4f}"
)

Output:

Probability of superiority: 0.4941
Rank-biserial correlation: -0.0119

The estimated probability of superiority is 49.41%.

That is extremely close to the 50% benchmark for balanced relative ordering, even though the means differ by approximately $274.

The rank-biserial effect size is also very close to zero.

This is why reporting an effect size alongside a p-value is so valuable. Statistical significance can change dramatically with sample size, while the effect-size estimate describes the observed magnitude on a relevant scale.

But what about the medians?

The sample median for group F exceeds that for group G by $101.50.

That is not inconsistent with the Mann–Whitney result. Differences in medians, means, and pairwise ordering are distinct properties of distributions.

If median price is genuinely the target, we can estimate its difference directly and attach uncertainty using bootstrap resampling.

import numpy as np

rng = np.random.default_rng(2026)
resamples = 2000

median_differences = np.empty(resamples)

for i in range(resamples):
    a_boot = rng.choice(
        group_a,
        size=len(group_a),
        replace=True
    )
    b_boot = rng.choice(
        group_b,
        size=len(group_b),
        replace=True
    )

    median_differences[i] = (
        np.median(a_boot) - np.median(b_boot)
    )

lower, upper = np.percentile(
    median_differences,
    [2.5, 97.5]
)

print(f"Observed median difference: $101.50")
print(f"Bootstrap 95% interval: [{lower:.2f}, {upper:.2f}]")

With this seed and dataset, the percentile bootstrap interval is approximately $9.49 to $201.51.

It is a further reminder that a median-difference interval and a Mann–Whitney p-value need not tell the same story: they concern different statistical targets.

The bootstrap interval also relies on the observations being a representative, suitably independent sample from the populations of interest. Resampling does not repair biased sampling or confounding.

7. Wilcoxon signed-rank: when observations come in pairs

The Mann–Whitney test is designed for independent groups. It is not appropriate when each observation in one group has a meaningful counterpart in the other.

Consider an infrastructure team measuring API latency before and after a performance optimization.

Suppose we have 24 endpoints, and each endpoint is measured under both configurations.

Our data structure looks like this:

The first two illustrative rows show the structure of paired data, not the exact generated values in the reproducible experiment below.

For each endpoint, the important quantity is:

Difference = After − Before

A negative difference indicates improved latency.

The baseline latency distribution might be heavily skewed, but the signed-rank test operates on the paired differences, not directly on the baseline measurements.

Step 1: Generate reproducible paired observations

We’ll simulate baseline latencies with a lognormal distribution and add an approximately symmetric change to each endpoint.

This synthetic example is intentionally separate from the real diamond analysis: the diamond records do not contain legitimate before-and-after pairs.

import numpy as np
from scipy import stats

rng = np.random.default_rng(2026)

n_endpoints = 24

before = np.round(
    rng.lognormal(
        mean=np.log(300),
        sigma=0.55,
        size=n_endpoints
    ),
    2
)

after = np.round(
    before + rng.normal(
        loc=-18,
        scale=20,
        size=n_endpoints
    ),
    2
)

# Two endpoints show no measured change.
after[[2, 5]] = before[[2, 5]]

differences = np.round(after - before, 2)

print(f"Endpoints: {len(differences)}")
print(f"Mean change: {np.mean(differences):.2f} ms")
print(f"Median change: {np.median(differences):.2f} ms")
print(f"Unchanged: {np.sum(differences == 0)}")
print(f"Improved: {np.sum(differences < 0)}")

The generated data contain 24 endpoint pairs, including two unchanged measurements and 16 endpoints with reduced latency.

The average observed change is approximately −9.98 milliseconds.

Step 2: Compare the paired t-test and Wilcoxon

paired_t = stats.ttest_rel(
    after,
    before,
    alternative="two-sided"
)

signed_rank = stats.wilcoxon(
    differences,
    alternative="two-sided",
    zero_method="wilcox",
    method="auto"
)

print(f"Paired t-test p-value: {paired_t.pvalue:.6f}")
print(f"Wilcoxon p-value: {signed_rank.pvalue:.6f}")
print(f"Wilcoxon statistic: {signed_rank.statistic:.1f}")

Results:

Paired t-test p-value: 0.020690
Wilcoxon p-value: 0.036257
Wilcoxon statistic: 62.0

Both tests reject their corresponding null hypotheses at the 5% threshold.

However, the meanings differ:

• The paired t-test concerns the population mean of the paired differences.

• The Wilcoxon signed-rank test assesses whether the distribution of differences is consistent with symmetry about zero.

• When the differences follow an appropriately symmetric location-shift distribution, Wilcoxon is commonly interpreted as a test of a zero median difference.

That last qualification matters.

Wilcoxon is not a universal test of whether the median paired difference equals zero under arbitrary asymmetric distributions.

Step 3: Understand how signed ranks work

For a simplified set of paired differences:

-12
 -8
 -3
  0
 +2
 +5

The default signed-rank procedure excludes zero differences, ranks the absolute values of the remaining differences, and then uses their signs to calculate rank sums.

The negative observations have absolute magnitudes 12, 8, and 3. The positive observations have magnitudes 2 and 5.

The test therefore incorporates information about both direction and relative magnitude.

A sign test, by contrast, uses the directions of nonzero differences without considering their magnitudes.

The sign test can be an alternative when the symmetry assumption for signed ranks is unsuitable and the primary target is the median direction of change.

Step 4: Handle ties, zeros, and rounding carefully

SciPy exposes three zero-handling conventions through the zero_method argument.

These methods can produce different results, especially in small datasets with numerous unchanged observations.

Also, be careful with floating-point subtraction.

If measurements are recorded to two decimal places, their differences should not accidentally acquire artificial distinctions from floating-point representation.

This is why the example computes and rounds the differences before passing them to SciPy.

Another critical rule: never independently remove missing values from before and after arrays.

Doing so can destroy the pairing.

Instead, construct a shared validity mask and remove entire incomplete pairs.

valid = np.isfinite(before) & np.isfinite(after)

clean_before = before[valid]
clean_after = after[valid]

differences = np.round(
    clean_after - clean_before,
    2
)

result = stats.wilcoxon(
    differences,
    zero_method="wilcox",
    method="auto"
)

print(result)

In a real experiment, also verify that measurements correspond to the same endpoints and comparable operating conditions.

Pairing is a property of the data collection design, not something that can be created by sorting both arrays.

8. Kruskal–Wallis: extending the analysis to multiple groups

So far, we’ve examined two-group comparisons.

What if there are five alternatives?

For example, our diamond dataset includes five cut categories:

• Fair

• Good

• Very Good

• Premium

• Ideal

We could run separate Mann–Whitney tests for every pair, but that creates ten comparisons.

Testing many hypotheses at a fixed 5% significance level increases the chance of obtaining at least one false positive.

Kruskal–Wallis offers a natural omnibus analysis.

Step 1: Prepare the five groups

We’ll reuse real diamond prices, this time grouped by cut.

import numpy as np
from scipy import stats
from plotnine.data import diamonds

cut_levels = [
    "Fair",
    "Good",
    "Very Good",
    "Premium",
    "Ideal"
]

samples = [
    diamonds.loc[
        diamonds["cut"] == cut,
        "price"
    ].to_numpy()
    for cut in cut_levels
]

for cut, values in zip(cut_levels, samples):
    print(
        f"{cut:10s} "
        f"n={len(values):5d}, "
        f"median=${np.median(values):,.2f}"
    )

The median prices vary considerably:

Notice that diamonds with an Ideal cut have the lowest median price in this dataset.

This does not mean that better cut quality makes a diamond cheaper.

Diamond price is strongly related to other features, particularly carat weight. The groups may have different distributions of those characteristics.

Kruskal–Wallis cannot eliminate such confounding.

Step 2: Run the omnibus test

from scipy import stats

result = stats.kruskal(*samples)

print(f"H-statistic: {result.statistic:.4f}")
print(f"p-value: {result.pvalue:.6e}")

Output:

H-statistic: 978.6208
p-value: 1.533415e-210

The p-value is extraordinarily small.

We reject the null hypothesis of identical underlying distributions across the five groups.

But there are two important restrictions on interpretation.

First, the test does not identify which specific groups differ.

Second, without suitable similarity in distributional shape, the result should not be described simply as evidence that the population medians differ.

Step 3: Calculate an approximate effect size

With more than 50,000 observations, even comparatively modest systematic differences can produce tiny p-values.

An approximate rank-based effect-size measure for Kruskal–Wallis is epsilon-squared:

Epsilon-squared = (H − k + 1) / (N − k)

Here:

• H is the Kruskal–Wallis test statistic.

• k is the number of groups.

• N is the total sample size.

We can calculate it directly.

n_total = sum(len(group) for group in samples)
k = len(samples)

epsilon_squared = (
    result.statistic - k + 1
) / (n_total - k)

print(f"Epsilon-squared: {epsilon_squared:.4f}")

Output:

Epsilon-squared: 0.0181

The rank-based effect-size estimate is approximately 0.018.

That is relatively small despite the overwhelming statistical significance.

This contrast illustrates a core principle of empirical analysis:

Statistical significance, effect magnitude, and practical importance are different concepts.

A business decision should consider all three.

9. Post-hoc testing: finding which groups differ

Rejecting the Kruskal–Wallis null hypothesis does not tell us where the differences occur.

One possible follow-up is to run pairwise Mann–Whitney tests with correction for multiple comparisons.

Dunn’s test is another common rank-based post-hoc procedure, but pairwise Mann–Whitney comparisons are useful here because they reuse an API we’ve already studied.

For an exploratory analysis, we’ll use SciPy’s false_discovery_control function with the Benjamini–Yekutieli method.

This adjustment is more conservative than Benjamini–Hochberg, but it can control the false discovery rate under arbitrary dependence among valid p-values.

Run all ten comparisons

import numpy as np
from itertools import combinations
from scipy import stats
from plotnine.data import diamonds

cut_levels = [
    "Fair",
    "Good",
    "Very Good",
    "Premium",
    "Ideal"
]

pairs = list(combinations(cut_levels, 2))

p_values = []

for cut_a, cut_b in pairs:
    a = diamonds.loc[
        diamonds["cut"] == cut_a,
        "price"
    ].to_numpy()

    b = diamonds.loc[
        diamonds["cut"] == cut_b,
        "price"
    ].to_numpy()

    result = stats.mannwhitneyu(
        a,
        b,
        alternative="two-sided",
        method="asymptotic"
    )

    p_values.append(result.pvalue)

p_values = np.asarray(p_values)

adjusted = stats.false_discovery_control(
    p_values,
    method="by"
)

for (cut_a, cut_b), raw, corrected in zip(
    pairs,
    p_values,
    adjusted
):
    print(
        f"{cut_a:10s} vs {cut_b:10s} "
        f"raw={raw:.5g} "
        f"adjusted={corrected:.5g}"
    )

Consider the comparison between Good and Very Good cuts.

The unadjusted p-value is below 0.05, but the adjusted result is above it.

Our conclusion changes again, this time because we’re acknowledging the family of hypotheses that was examined.

This is particularly important in product analytics, where teams may compare dozens of customer segments, feature variants, time windows, and engagement measures.

Finding one attractive p-value among many tests is not the same as obtaining strong confirmatory evidence.

For strictly confirmatory analyses, specify the primary hypothesis and multiplicity strategy before examining the outcomes.

Documentation: SciPy’s false-discovery-rate correction.

10. Small samples: exact tests, ties, and permutation methods

Large samples often allow asymptotic approximations.

Small samples deserve more care.

SciPy’s Mann–Whitney implementation supports different p-value calculation methods:

• auto selects a method using sample characteristics.

• exact calculates an exact null distribution under its applicable assumptions.

• asymptotic uses a large-sample approximation with tie correction.

• A PermutationMethod instance requests a permutation-based calculation.

An important limitation is that the Mann–Whitney exact method does not correct for ties.

If your dataset contains tied observations and small samples, a permutation-based approach may be more appropriate.

Example: two small order-value samples

import numpy as np
from scipy import stats

store_a = np.array(
    [12, 12, 15, 18, 21, 25],
    dtype=float
)

store_b = np.array(
    [10, 14, 14, 19, 20, 24],
    dtype=float
)

permutation_config = stats.PermutationMethod(
    n_resamples=9999,
    rng=np.random.default_rng(7)
)

result = stats.mannwhitneyu(
    store_a,
    store_b,
    alternative="two-sided",
    method=permutation_config
)

print(f"U-statistic: {result.statistic:.1f}")
print(f"Permutation p-value: {result.pvalue:.6f}")

Output:

U-statistic: 19.0
Permutation p-value: 0.902597

Here, the sample is small and includes repeated values.

The permutation procedure evaluates the statistic relative to rearrangements of observations under the null hypothesis.

This avoids relying solely on the asymptotic approximation, but it still requires a valid exchangeability assumption for the permutation design.

Permutation tests are not automatically valid for dependent observations, confounded treatment assignment, or arbitrary differences in distributional shape.

The random seed makes the result reproducible when sampling permutations; for these small groups, the configured budget is sufficient for exhaustive enumeration.

The general lesson is to pay attention to how SciPy calculates a p-value, not merely which test function was called.

11. Common mistakes that invalidate nonparametric analyses

The most serious statistical errors often happen before the hypothesis test.

Mistake 1: Treating every non-normal sample as a Mann–Whitney problem

Suppose your business cares about average customer spending.

A Mann–Whitney test cannot simply replace a test of means without changing the inferential target.

Better options may include Welch’s t-test, a prespecified robust mean-based estimator, a suitably designed permutation test, or bootstrap confidence intervals.

SciPy also supports trimmed versions of the independent t-test.

For example:

from scipy import stats
from plotnine.data import diamonds

a = diamonds.loc[
    diamonds["color"] == "F", "price"
].to_numpy()

b = diamonds.loc[
    diamonds["color"] == "G", "price"
].to_numpy()

result = stats.ttest_ind(
    a,
    b,
    equal_var=False,
    trim=0.10
)

print(f"Trimmed t-statistic: {result.statistic:.4f}")
print(f"p-value: {result.pvalue:.6g}")

The trimmed t-test is an implementation of a Yuen-style test that reduces the influence of extreme observations.

However, trimming changes the target from the ordinary population mean to a trimmed-mean comparison.

That distinction matters if total revenue or average revenue per user is the primary business objective.

Mistake 2: Forgetting independence

Suppose one customer contributes 50 transactions and another contributes only one.

Treating all 51 transactions as independent customer-level observations is usually inappropriate.

The transactions from the same customer may be correlated.

Rank-based tests do not automatically solve clustering or repeated-measurement problems.

Depending on the experiment, better strategies include customer-level aggregation, cluster-aware resampling, hierarchical modeling, or other methods that reflect the sampling and assignment design.

The experimental unit should be defined before analysis.

Mistake 3: Choosing the test after seeing the p-values

Running several tests and reporting whichever yields significance creates selection bias.

A more defensible process is to declare:

• The primary metric and estimand.

• The experimental unit and assignment mechanism.

• The hypothesis and significance threshold.

• The primary statistical test.

• The handling of missing values, ties, and outliers.

• The multiple-testing adjustment, if relevant.

Sensitivity analyses can still be valuable, but they should be labeled as such rather than quietly substituted for the primary analysis.

Mistake 4: Equating a small p-value with a useful effect

Our Kruskal–Wallis example illustrates this perfectly.

Its p-value was approximately 1.53 × 10⁻²¹⁰, yet the approximate epsilon-squared effect size was only 0.018.

Large samples can provide very strong evidence for differences that may be small in practical terms.

Whenever possible, report effect estimates and uncertainty intervals.

Mistake 5: Ignoring confounding

The real diamond dataset is observational.

Color groups might differ systematically in carat weight, clarity, or cut.

Testing whether group prices differ does not tell us what caused those differences.

A regression model or a carefully designed matched analysis may help investigate adjusted associations, but causal interpretation requires appropriate assumptions and study design.

For genuine A/B experiments, randomization is a major advantage because it reduces systematic confounding in expectation.

Mistake 6: Assuming a normality test should choose the analysis

A common workflow is to run a normality test first, then choose between a t-test and a rank-based test depending on its p-value.

This approach can be misleading.

At large sample sizes, normality tests can detect minor departures that have little practical effect on the reliability of a mean-based analysis.

At small sample sizes, they may have little power to identify consequential non-normality.

In addition, choosing the primary hypothesis test from the same data can alter the statistical properties of the resulting procedure.

Use descriptive summaries, subject-matter knowledge, the data-generating mechanism, and the intended estimand to select an analysis.

Normality diagnostics can provide useful context, but they should not function as an automatic statistical decision tree.

12. A practical decision framework

Use the following workflow when designing your next analysis.

Choosing a test

This is a design guide, not a substitute for checking independence, distributional assumptions, and the desired estimand.

For more than two independent groups when means are the target, modern SciPy also supports Welch’s ANOVA through scipy.stats.f_oneway with equal_var=False.

For more complex designs, such as repeated observations across three or more conditions, nested measurements, covariate adjustment, or clustered experiments, these simple tests may not be sufficient.

A good analysis starts with a clear representation of how the data were collected.

13. What to report in an empirical study

A useful statistical report should allow a reader to understand both the evidence and its limitations.

For a two-group comparison, report the sample sizes, descriptive statistics, chosen test, test statistic, p-value, effect-size estimate, and a relevant uncertainty interval.

Explain what parameter or distributional feature the test evaluates.

For paired measurements, explain how pairs were constructed and what was done with missing observations and zero differences.

For multiple groups, distinguish the omnibus result from post-hoc comparisons and state how multiplicity was handled.

For observational data, explicitly discuss confounding and avoid causal language.

A good result section might say:

The mean price differed between color grades F and G under Welch’s t-test (p ≈ 4.57 × 10⁻⁷), with an estimated mean difference of −$274.25. A Mann–Whitney comparison of relative ordering was not significant at the 5% level (p ≈ 0.140; estimated probability of superiority 0.494). Because the observations were not randomized and the tests target different population features, these results should not be interpreted as contradictory causal findings.

This communicates far more than either “significant” or “not significant” alone.

14. Final takeaways

The three rank-based tests covered in this tutorial are essential tools, but their value depends on understanding their statistical targets.

Mann–Whitney U is useful for comparing two independent groups through ranks and relative ordering. It is not universally equivalent to a test of medians.

Wilcoxon signed-rank is useful for related observations when the distribution of paired differences makes its symmetry-based framework appropriate. Pairing must reflect the actual experimental design.

Kruskal–Wallis provides an omnibus rank-based test for multiple independent groups. It does not identify individual group differences and does not replace post-hoc testing or consideration of effect sizes.

Most importantly, nonparametric tests are not automatic repairs for failed normality assumptions.

Our real-data example demonstrated why. Welch’s test found strong evidence of a difference in mean diamond prices between two color grades, while Mann–Whitney found no significant evidence in their relative ordering at the chosen threshold. Both outcomes can be statistically coherent because the tests examine different properties.

The strongest statistical workflow chooses the estimand first, respects the sampling design, uses appropriate inference, and reports effect sizes and uncertainty rather than treating a single p-value as the final answer.

Your next step: Run the examples locally, then repeat the independent-group comparison using another pair of diamond grades. Before viewing the p-values, write down whether your question concerns mean price, median price, or relative ordering. Compare the results, inspect the effect sizes, and see whether your initial research question—not the most attractive p-value—determines the conclusion.