Rules vs model bake-off: measuring when learned models actually earn their complexity

A machine learning model achieves 77.5% accuracy. A handful of hand-written rules achieves 76.0%. The model wins the benchmark.

Should you deploy it?

Not necessarily.

That extra 1.5 percentage points might prevent thousands of costly mistakes. Or it might introduce more infrastructure, slower predictions, and additional maintenance work than the improvement is worth.

The difference between a technically superior model and a commercially superior system is where many machine learning projects succeed or fail.

In this tutorial, we’ll conduct a reproducible bake-off between three approaches:

• Hand-written rules: Deterministic conditions created from domain knowledge.

• RuleFit: A hybrid approach that extracts decision rules from trees and selects useful rules with a sparse linear model.

• Gradient boosting: A learned ensemble that captures nonlinear relationships and feature interactions.

We’ll evaluate all three on the same synthetic warehouse-delivery dataset, using identical training, validation, and test periods.

Our comparison goes beyond predictive accuracy. We’ll measure inference latency, training time, error costs, explainability, and the maintenance burden associated with putting each approach into production.

By the end, you’ll have both a working benchmark and a practical framework for deciding when a learned model genuinely earns its complexity.

1. Why model accuracy alone is an incomplete decision criterion

Imagine an e-commerce company processing thousands of orders every day.

Its operations team wants to identify orders at risk of late delivery so that employees can intervene before customers are affected.

An experienced warehouse manager knows several warning signs:

• A warehouse queue is unusually long.

• Worker utilization is approaching capacity.

• Heavy rainfall coincides with a long delivery route.

• A promotional campaign creates additional demand.

• Weekend staffing patterns make congestion harder to resolve.

These observations naturally translate into deterministic rules.

A data scientist proposes a predictive model instead. The model might discover that moderate rainfall becomes problematic only when utilization exceeds a particular threshold, or that promotional demand affects certain delivery distances differently.

Both approaches make sense.

The essential question is whether the learned relationships create enough additional value to justify their operational costs.

Four dimensions of model quality

A serious model comparison should consider at least four dimensions.

These dimensions interact.

For example, a slightly less accurate model might be easier to correct when business policies change. Conversely, a more sophisticated model could reduce enough costly exceptions to pay for its entire maintenance budget.

Complexity is justified when its incremental benefits exceed its incremental costs—not merely when a test metric increases.

2. The experiment: predicting late warehouse deliveries

We’ll create a synthetic dataset representing 16,000 warehouse orders.

Each row contains six features available before the delivery-risk decision.

The target is binary:

• 0: Delivery is not late.

• 1: Delivery is late.

Rather than constructing a perfectly deterministic target, we’ll generate late deliveries probabilistically.

That matters because real operational outcomes contain randomness. Two deliveries with identical recorded features can have different outcomes due to unobserved factors.

We’ll also introduce a small distribution shift near the end of the dataset: increasing queue lengths and rainfall severity.

This gives us a more meaningful deployment-like test than shuffling every observation randomly.

Our chronological split

The test set is never used to fit models or choose decision thresholds.

This distinction prevents a common mistake: reporting an apparently excellent test score after repeatedly using that same test set to optimize the system.

One important limitation remains: our dataset is synthetic, so this experiment demonstrates methodology, not a universal ranking of algorithms. We know its generating process, even though the models do not.

A real deployment would also need genuine historical labels, feature-availability checks, and evaluation across multiple time periods.

3. Set up a reproducible benchmarking environment

We’ll use Python and recent versions of NumPy, SciPy, and scikit-learn.

The benchmark was tested with:

• Python 3.13

• NumPy 2.3.5

• SciPy 1.17.0

• scikit-learn 1.8.0

These versions are suitable for reproducing the measurements below. As of October 2026, scikit-learn 1.9.1 is also available, but runtime measurements can change between releases. The relevant estimator interfaces remain supported.

Create an isolated environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install numpy==2.3.5 scipy==1.17.0 scikit-learn==1.8.0

The activation command assumes a POSIX-compatible shell.

We’ll implement the experiment in a single file named bakeoff.py. There are no external datasets or API keys.

Why implement the RuleFit comparison ourselves?

The maintained imodels project provides a full RuleFitClassifier. Its October 2026 release supports scikit-learn-style fitting and prediction.

For this benchmark, we’ll use a smaller, inspectable RuleFit-style implementation built entirely with scikit-learn.

It reproduces the central RuleFit idea:

1. Train a tree ensemble.

2. Extract Boolean features representing decision-tree paths.

3. Combine those features with the original numeric features.

4. Fit a sparse linear model that selects useful terms.

Our variant uses terminal-leaf paths and standardized original features. It does not reproduce every detail of the full Friedman–Popescu algorithm, such as its complete candidate-rule construction and winsorization procedure.

That distinction matters: the results measure this explicit RuleFit variant, not the packaged implementation or every possible RuleFit configuration.

The advantage is that we can examine every part of the model without introducing another dependency.

4. Build the three competing approaches

Approach A: deterministic hand-written rules

The simplest system begins with domain knowledge.

For example:

• Add two risk points when the queue exceeds 72 orders.

• Add 1.5 points when utilization exceeds 83%.

• Add 1.5 points when severe rainfall coincides with a long route.

• Add a point for an active promotion.

• Add a smaller adjustment for weekend congestion.

These are deliberately imperfect heuristics.

The rules do not know the true probabilities used to generate the targets. Their thresholds are approximations of operational knowledge.

This is important for a fair comparison. If we copied the target-generating formula directly into the rule system, we’d give it information unavailable to a real warehouse manager.

Approach B: sparse learned rules

Our RuleFit variant uses gradient-boosted decision trees to discover candidate conditions.

Consider an illustrative learned tree:

Illustrative tree structure, not an extracted result from the fitted model.Each terminal path becomes a Boolean feature. One might represent a condition equivalent to an unusually long queue combined with high utilization.

A sparse logistic regression then assigns coefficients to these features. L1 regularization encourages many coefficients to become zero.

Conceptually:

Prediction score = intercept + linear feature contributions + selected rule contributions.

This combines learned interactions with an additive model whose surviving terms can be inspected.

However, hundreds of candidate rules may still produce dozens of nonzero terms. Sparse does not automatically mean simple enough for a human to audit.

Approach C: histogram-based gradient boosting

Our third approach uses HistGradientBoostingClassifier.

Rather than extracting rules and training a separate sparse selector, this model directly optimizes an ensemble of trees.

Histogram-based boosting discretizes feature values into bins, making tree construction computationally efficient on suitable datasets.

Its strengths include:

• Learning nonlinear relationships.

• Capturing conditional interactions.

• Handling continuous features without manually creating every threshold.

• Strong predictive performance on many structured datasets.

The disadvantages are mainly operational and interpretive. Individual predictions depend on contributions from multiple trees, and the system must manage trained model artifacts and data-dependent behavior.

The class is part of scikit-learn’s supported API.

5. The complete runnable head-to-head benchmark

The following script generates the data, fits the models, selects thresholds using validation data, evaluates the untouched test set, and measures inference latency.

It also computes a paired bootstrap confidence interval for the accuracy difference between hand rules and boosting.

Save the complete script as bakeoff.py.

import time

import numpy as np
from scipy import sparse
from sklearn.ensemble import (
    GradientBoostingClassifier,
    HistGradientBoostingClassifier,
)
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, f1_score
from sklearn.preprocessing import OneHotEncoder, StandardScaler


SEED = 42
rng = np.random.default_rng(SEED)

# --------------------------------------------------
# 1. Generate synthetic warehouse deliveries
# --------------------------------------------------

n = 16_000

queue = rng.uniform(10, 110, n)
util = rng.uniform(0.35, 1.0, n)
distance = rng.uniform(1, 80, n)
rain = rng.uniform(0, 1, n)
promo = rng.binomial(1, 0.25, n)
weekend = rng.binomial(1, 0.29, n)

# Simulate a later period with heavier demand.
queue[12_000:] = np.clip(queue[12_000:] + 8, 10, 110)
rain[12_000:] = np.clip(rain[12_000:] + 0.08, 0, 1)

X = np.column_stack(
    (queue, util, distance, rain, promo, weekend)
).astype("float64")

# Latent late-delivery probability.
logit = (
    -4.3
    + 1.9 * (queue > 70)
    + 1.3 * (util > 0.78)
    + 1.2 * ((rain > 0.75) & (distance > 40))
    + 0.55 * promo
    + 0.65 * ((weekend == 1) & (queue > 60))
    + 0.02 * distance
    + 0.008 * queue
    + 1.5 * np.maximum(0, util - 0.8) * distance / 100
)

prob = 1 / (1 + np.exp(-logit))
y = rng.binomial(1, prob)

# Chronological train, validation, and test splits.
X_train, X_val, X_test = (
    X[:9_600],
    X[9_600:12_000],
    X[12_000:],
)
y_train, y_val, y_test = (
    y[:9_600],
    y[9_600:12_000],
    y[12_000:],
)


# --------------------------------------------------
# 2. Hand-written risk rules
# --------------------------------------------------

def rule_score(a):
    q, u, d, r, p, w = a.T

    return (
        2.0 * (q >= 72)
        + 1.5 * (u >= 0.83)
        + 1.5 * ((r >= 0.70) & (d >= 40))
        + 1.0 * p
        + 0.75 * ((w >= 0.5) & (q >= 50))
    )


# --------------------------------------------------
# 3. Compact RuleFit-style estimator
# --------------------------------------------------

class RuleFitLite:
    """
    Gradient-boosted leaf paths become Boolean rules.
    L1 logistic regression selects useful rule and
    original-feature contributions.
    """

    def __init__(self):
        self.generator = GradientBoostingClassifier(
            n_estimators=70,
            learning_rate=0.06,
            max_depth=2,
            subsample=0.8,
            random_state=SEED,
        )

        self.encoder = OneHotEncoder(
            handle_unknown="ignore",
            sparse_output=True,
        )

        self.scaler = StandardScaler()

        self.selector = LogisticRegression(
            l1_ratio=1.0,
            solver="liblinear",
            C=0.07,
            max_iter=2000,
            random_state=SEED,
        )

    def _leaves(self, a):
        return self.generator.apply(a).reshape(len(a), -1)

    def _design(self, a):
        linear = sparse.csr_matrix(
            self.scaler.transform(a)
        )

        rules = self.encoder.transform(
            self._leaves(a)
        )

        return sparse.hstack(
            [linear, rules],
            format="csr",
        )

    def fit(self, a, target):
        self.generator.fit(a, target)
        self.scaler.fit(a)
        self.encoder.fit(self._leaves(a))

        self.selector.fit(
            self._design(a),
            target,
        )

        return self

    def predict_proba(self, a):
        return self.selector.predict_proba(
            self._design(a)
        )


# --------------------------------------------------
# 4. Define the competitors
# --------------------------------------------------

predictions = {}

models = {
    "Hand rules": (
        None,
        rule_score,
    ),
    "RuleFit (compact)": (
        RuleFitLite(),
        None,
    ),
    "Hist. boosting": (
        HistGradientBoostingClassifier(
            learning_rate=0.06,
            max_iter=150,
            max_leaf_nodes=15,
            l2_regularization=1.0,
            random_state=SEED,
        ),
        None,
    ),
}


# --------------------------------------------------
# 5. Tune thresholds on validation data only
# --------------------------------------------------

def threshold_for_acc(scores, target):
    candidates = np.unique(
        np.quantile(
            scores,
            np.linspace(0, 1, 101),
        )
    )

    default = 0.5 if np.max(scores) <= 1 else 2.0

    candidates = np.unique(
        np.r_[candidates, default]
    )

    accuracies = [
        accuracy_score(target, scores >= cutoff)
        for cutoff in candidates
    ]

    return float(
        candidates[int(np.argmax(accuracies))]
    )


# --------------------------------------------------
# 6. Evaluate accuracy and inference latency
# --------------------------------------------------

print(
    "Class prevalence:",
    [round(z.mean(), 3) for z in (y_train, y_val, y_test)],
)

for name, (model, score_fn) in models.items():
    started = time.perf_counter()

    if model is not None:
        model.fit(X_train, y_train)

        def score(a, fitted_model=model):
            return fitted_model.predict_proba(a)[:, 1]

    else:
        score = score_fn

    fit_seconds = time.perf_counter() - started

    cutoff = threshold_for_acc(
        score(X_val),
        y_val,
    )

    test_scores = score(X_test)
    y_pred = (test_scores >= cutoff).astype(int)

    predictions[name] = y_pred

    # Warm up inference.
    batch = X_test[:128]

    for _ in range(10):
        score(batch)

    # Measure microseconds per row for 128-row batches.
    measurements = []

    for _ in range(60):
        start = time.perf_counter()
        score(batch)

        elapsed_us = (
            time.perf_counter() - start
        ) * 1_000_000

        measurements.append(
            elapsed_us / len(batch)
        )

    accuracy = accuracy_score(y_test, y_pred)
    f1 = f1_score(y_test, y_pred)
    latency = np.median(measurements)

    print(
        name,
        "threshold:", round(cutoff, 4),
        "accuracy:", round(accuracy, 4),
        "f1:", round(f1, 4),
        "fit_seconds:", round(fit_seconds, 3),
        "us_per_row_128:", round(latency, 2),
        "confusion:", confusion_matrix(
            y_test,
            y_pred,
        ).tolist(),
    )

    if name == "RuleFit (compact)":
        coefs = model.selector.coef_[0]
        n_features = X_train.shape[1]

        selected_rules = np.count_nonzero(
            coefs[n_features:]
        )

        selected_linear = np.count_nonzero(
            coefs[:n_features]
        )

        print(
            "RuleFit selected:",
            selected_rules,
            "rules and",
            selected_linear,
            "linear terms",
        )


# --------------------------------------------------
# 7. Paired bootstrap: boosting minus hand rules
# --------------------------------------------------

boost_correct = (
    predictions["Hist. boosting"] == y_test
).astype(int)

rules_correct = (
    predictions["Hand rules"] == y_test
).astype(int)

paired_delta = boost_correct - rules_correct

boot_rng = np.random.default_rng(99)

bootstrap = [
    paired_delta[
        boot_rng.integers(
            0,
            len(y_test),
            len(y_test),
        )
    ].mean()
    for _ in range(2000)
]

lower, upper = np.quantile(
    bootstrap,
    [0.025, 0.975],
)

print(
    "Accuracy lift:",
    round(paired_delta.mean(), 4),
    "95% bootstrap interval:",
    round(lower, 4),
    round(upper, 4),
)

Run it using a single thread for the main numerical libraries to reduce one source of timing variation.

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 python bakeoff.py

Why this benchmark is reasonably fair

Every approach receives the same six input features and is evaluated on exactly the same test observations.

The learned models use the same training window.

Decision thresholds are selected to maximize validation accuracy using the same threshold-search procedure. The hand-written rules receive the same opportunity to select an operating threshold.

Neither learned model gets to see test labels during training.

There are still limitations:

• The model hyperparameters are fixed rather than exhaustively optimized.

• The hand-written rule coefficients are fixed, although the final cutoff is tuned.

• The comparison excludes data ingestion, service overhead, network requests, and serialization.

• One synthetic chronological holdout cannot establish general performance across real warehouses.

• Our compact RuleFit implementation is a specific variant, not a benchmark of every possible rule-learning algorithm.

These constraints make the experiment an informative head-to-head, not a definitive ranking.

6. Results: the models do improve accuracy, but by how much?

The following results came from executing the benchmark using the environment specified above.

One honest comparison table

How to interpret this table: Accuracy, F1, rule counts, and latency are experimental measurements. Maintenance hours are illustrative engineering-planning assumptions, not observations from a deployed system.

Inference timings represent the median amortized cost per order from 60 calls using batches of 128 rows. They are not single-request p95 or p99 service latencies, and different machines will produce different numbers.

The maintenance estimates assume an existing deployment platform, a shared feature pipeline, periodic performance reviews, and occasional changes. If those prerequisites do not exist, the absolute costs can be substantially higher.

The accuracy gains

Test accuracy on 4,000 held-out orders:

The difference from hand rules to boosting is about 1.58 percentage points.

In a test set of 4,000 deliveries, that corresponds to approximately 63 additional correct predictions.

The compact RuleFit model captures nearly all the measured accuracy improvement achieved by boosting. However, boosting has a higher F1 score, meaning the methods differ in their positive-class error trade-offs.

And notably, RuleFit is slower in this particular implementation.

That may seem surprising, but prediction speed depends heavily on implementation architecture.

Our RuleFit variant invokes a tree ensemble, extracts leaf assignments, performs sparse encoding, and evaluates a logistic model. Histogram boosting uses a more specialized prediction implementation.

An interpretable model is not necessarily a fast model.

Does the accuracy difference appear to be noise?

We also calculate a paired bootstrap confidence interval using the test observations.

For boosting versus hand rules, the observed improvement is approximately:

• Accuracy lift: 1.58 percentage points.

• Approximate 95% paired bootstrap interval: 0.60 to 2.62 percentage points.

Because the interval is above zero, this particular result supports a genuine performance difference on the simulated test distribution.

But there are qualifications.

The bootstrap treats the evaluated observations as independent resampling units. Real orders may exhibit temporal dependence—for example, many orders being affected by the same operational incident.

In that setting, a block bootstrap or multiple chronological evaluation windows would be preferable.

Statistical evidence of a predictive improvement also does not imply that the improvement is economically worthwhile.

7. Examine the mistakes, not just accuracy

Accuracy combines correct positive and negative decisions into one number.

Operational systems care about which mistakes happen.

For the test set, the confusion matrices yield:

A false negative means an order will be late, but the system did not flag it.

A false positive means the system raised an alert for an order that was not late.

Compare the hand rules with boosting.

Boosting detects 104 additional late deliveries but produces 41 additional false alerts.

Whether this is an excellent or terrible trade depends on what happens after an alert.

Introducing asymmetric business costs

Suppose the warehouse estimates:

• A missed late-delivery warning costs $15 in avoidable operational or customer-service consequences.

• An unnecessary alert costs $2 in staff attention and handling.

These are hypothetical business assumptions, not measured costs from the dataset.

We can define:

Expected error cost = (false negatives × 15) + (false positives × 2).

Using the measured confusion matrices:

Under this cost model, boosting saves $1,478 compared with hand rules for the 4,000 evaluated orders.

That’s a potentially meaningful improvement.

Notice, though, that these are costs at the validation-accuracy-optimized thresholds. A genuine cost-sensitive deployment should instead select thresholds on validation data to minimize expected cost. The resulting decisions and savings could differ.

Sensitivity to business assumptions

Suppose the operation handles 10,000 comparable orders per month.

At the original hypothetical cost assumptions, boosting’s measured advantage scales to approximately $3,695 per month before engineering and infrastructure expenses.

Now suppose both false positives and false negatives cost only $2.

In that scenario, the measured difference is:

• Hand rules: 962 errors.

• Boosting: 899 errors.

• Reduction: 63 errors per 4,000 orders.

• Benefit at 10,000 monthly orders: approximately $315.

If the learned model requires an additional six engineer-hours per month at an internal loaded cost of $100 per hour, the extra $600 maintenance expense would outweigh that $315 improvement.

The less accurate hand-written system would therefore be the better economic choice under those assumptions.

We have not changed the predictive ranking at all.

We changed what errors are worth.

That is exactly why model selection is a business optimization problem rather than an accuracy contest.

8. Explainability: counting rules is not enough

We should distinguish three different kinds of explanation.

Decision-level explanation

A hand-written rules engine can usually identify the exact clauses that fired.

For example, an order may have been flagged because its queue exceeded the configured threshold and its warehouse utilization was unusually high.

The operations team can verify those measurements and question the thresholds.

Model-level explanation

A sparse learned-rule model is additive, so its surviving rule coefficients can be inspected.

But our fitted implementation selected 53 rule terms and three linear terms from 280 candidate leaf-path indicators.

That’s significantly more complicated than five hand-written clauses.

Furthermore, the coefficients operate together. A positive coefficient does not necessarily mean a particular rule independently causes late delivery.

The model describes statistical associations used for prediction, not causal effects.

Ensemble-level explanation

Gradient boosting can be examined using techniques such as:

• Permutation feature importance on held-out data.

• Partial dependence plots.

• Individual conditional expectation plots.

• Tree-specific contribution methods where supported by the chosen implementation.

These can be extremely useful, but their explanations require careful interpretation, particularly when features are correlated.

Permutation importance, for example, reports how disrupting a feature affects a model’s performance. It does not establish that changing that feature in the real world would produce the same effect.

For operational deployment, explanation quality should be evaluated with actual users.

A model is not sufficiently explainable merely because an engineer can produce a chart. The relevant question is whether the chart helps someone investigate a mistake, approve a decision, or maintain a policy.

9. Maintenance cost: where model complexity becomes expensive

Training the model is only one part of the lifecycle.

Maintaining hand-written rules

Rules typically require:

• Monitoring error rates and changing conditions.

• Reviewing threshold changes.

• Testing rule precedence and interactions.

• Maintaining feature definitions.

• Preserving a change history and rollback capability.

Their advantage is that a requested policy change can often be applied directly.

Their weakness is accumulated complexity.

Five independent clauses may be easy to maintain. Five hundred interacting exceptions may not be.

A rules engine can acquire technical debt just as quickly as a machine learning pipeline.

Maintaining learned rules

RuleFit adds another set of responsibilities:

• Maintaining representative training labels.

• Regenerating candidate rules.

• Tracking the selected terms after retraining.

• Reviewing rule stability across versions.

• Validating model artifacts.

• Monitoring data and prediction drift.

Some of these activities are also required for boosted models.

The additional complexity is justified if learned rules provide enough predictive benefit while remaining meaningfully easier to investigate than a more opaque ensemble.

Our compact variant reveals a particular drawback: maintaining both a tree generator and a sparse selector creates more moving parts than either component alone.

Maintaining gradient boosting

Boosting requires a reliable pipeline for:

• Training and validation.

• Feature-schema compatibility.

• Model artifact versioning.

• Retraining and rollback.

• Monitoring performance after deployment.

• Handling changes in data availability and distribution.

But it does not automatically require more maintenance than every interpretable alternative.

A mature model-serving platform may make deploying a standard boosted estimator easier than maintaining a custom rule-extraction implementation.

This is why our illustrative maintenance estimates place boosting below RuleFit.

The operational ecosystem matters more than the simplistic assumption that a larger mathematical model must always be harder to maintain.

10. The cherry on the cake: Netflix’s million-dollar lesson

One of the most instructive published examples of accuracy failing to justify complexity comes from the Netflix Prize.

This is a historical case, documented by Netflix in 2012, rather than a claim about its current recommendation architecture.

Netflix launched a competition in 2006 to improve predictions of movie ratings.

Teams developed increasingly elaborate recommendation systems, culminating in a prize-winning ensemble that surpassed Netflix’s original accuracy target.

But Netflix did not deploy the final winning ensemble.

In its own engineering retrospective, Netflix explained that the additional offline accuracy did not justify the engineering effort necessary to bring the full solution into production.

Crucially, Netflix had already deployed useful components from earlier competition work, including matrix factorization and restricted Boltzmann machine techniques.

The outcome was not a rejection of machine learning. It was a rejection of additional complexity whose marginal value was insufficient.

Nor was it literally a case of hand-written rules defeating machine learning; the retained recommendation components were themselves learned algorithms.

The lesson is still directly applicable to our experiment.

A production system should not absorb the cost of an additional layer, model, or ensemble simply because an offline metric improves.

The improvement needs to survive operational constraints and produce worthwhile value.

A more directly rule-focused published result comes from the 2023 paper Simple rules outperform machine learning for personnel selection.

In that competition study (the 3rd annual SIOP machine learning competition), a simple, explainable rule-based model outperformed most of the machine learning models entered and was comparable to the top performers.

That study demonstrates the predictive competitiveness of simple rules, but it does not establish a long-term production maintenance-cost win.

Taken together, the two cases support a narrower, defensible conclusion: simple methods can be genuinely competitive, and an offline accuracy win is not sufficient evidence that a more complex solution should be deployed.

11. A practical deployment framework

We can translate the benchmark into concrete release criteria.

Gate 1: meaningful incremental improvement

Start with a strong rules baseline.

Do not accept a learned model merely because its metric is higher.

Require evidence that the improvement is:

• Measured on genuinely held-out data.

• Stable across relevant time periods and operational segments.

• Large enough to matter economically.

• Consistent with the kinds of mistakes the business wants to prevent.

A small improvement can be valuable at sufficient volume.

A large improvement can be useless if it affects outcomes that nobody can change.

Gate 2: operational feasibility

Evaluate the full system, not only the estimator.

Questions include:

• Are all required features available at decision time?

• Can the model handle missing or malformed inputs?

• Does inference meet the end-to-end latency budget?

• What happens when model loading or prediction fails?

• Can the previous version be restored quickly?

Our benchmark shows that the hand rules are extremely fast in vectorized form.

However, speed alone should not dictate the choice. Ten microseconds of amortized estimator inference may be negligible inside a service that spends tens of milliseconds retrieving features over a network.

Measure the complete request path before making infrastructure decisions.

Gate 3: explainability and auditability

For each candidate, document:

• What inputs influenced the decision.

• How a mistaken decision can be investigated.

• Who is responsible for approving changes.

• Whether the explanation is understandable to operational stakeholders.

• How conflicting policy requirements are handled.

For a small, high-transparency workflow, rules may be preferable even when a learned model improves accuracy.

For a high-volume detection task, the predictive improvement may justify a more complex explanation process.

Gate 4: expected net value

A useful decision equation is:

Net monthly value = avoided error cost − additional engineering cost − additional infrastructure cost − other operational costs.

The last category matters.

It can include alert fatigue, delayed investigations, customer consequences, retraining failures, or interruptions during deployment.

A proper cost model should also account for uncertainty rather than relying on one optimistic point estimate.

The benchmark’s paired confidence interval is a starting point for that discussion, not a complete economic risk analysis.

Gate 5: controlled rollout

Once a learned model passes offline evaluation, avoid immediately replacing the incumbent.

A safer deployment strategy is:

1. Keep the rules system as the decision-making baseline.

2. Run the learned model in shadow mode on incoming events.

3. Compare predictions when eventual ground-truth outcomes become available.

4. Validate operational latency and failure behavior.

5. Roll out gradually with documented rollback criteria.

Shadow mode is especially useful for exposing feature-timing mistakes and differences between historical datasets and live traffic.

If deploying a model changes the operational interventions being performed, its outcomes may also change. In that situation, passive prediction comparisons may not establish the true effect of the model. A properly designed controlled experiment may be necessary.

12. When each approach deserves to win

Choose hand-written rules when…

Domain constraints are stable and explicit, transparency is essential, updates are infrequent, or error costs are too small to justify the overhead of training and deployment.

Choose RuleFit when…

Learned interactions matter, a reasonably compact additive explanation has value, and the rule-extraction implementation is supportable. Verify the final rule count rather than assuming sparsity guarantees simplicity.

Choose gradient boosting when…

Nonlinear feature interactions produce material improvements, enough representative labels exist, and the engineering platform can support training, monitoring, explanations, and rollback.

These are decision criteria, not universal performance rankings.

A learned model can be cheaper to maintain than a sprawling rules engine. A simple rule system can be more robust than a frequently retrained model when the environment changes in ways the training data cannot anticipate.

The correct comparison must include those possibilities.

13. Extensions for a stronger real-world bake-off

Our example is deliberately small enough to understand and reproduce.

For a production-grade evaluation, I would extend it in several directions.

Use rolling time windows

Instead of relying on one chronological split, test over multiple successive periods.

Record the performance of each model separately.

A model that wins by two percentage points in one quarter but loses in the next may have an unstable advantage.

Evaluate important subgroups

Break results down by factors such as high-demand periods, route lengths, weather severity, and promotional events.

The overall winner may not be the best choice for every operational segment.

Avoid automatically building separate models for every group, however. Subgroup specialization introduces additional complexity and increases the risk of overfitting small samples.

Measure calibration and decision costs

Accuracy is not sufficient when decisions depend on probability thresholds.

Evaluate whether predicted probabilities reflect observed event frequencies, using measures such as Brier score and reliability diagrams.

The hand-rule score in our experiment is not a probability, so a direct calibration comparison would require defining and validating an appropriate probability-mapping procedure.

Select action thresholds using explicit cost assumptions on validation data and evaluate those fixed thresholds on the test set.

Measure realistic service latency

Use the intended deployment environment.

Measure p50, p95, and p99 end-to-end latency under representative concurrency, including feature retrieval, preprocessing, inference, and logging.

Memory consumption, cold-start time, and throughput may matter as much as per-row CPU time.

Include model ownership cost

Track actual engineer-hours devoted to:

• Changes and incident resolution.

• Retraining and validation.

• Feature maintenance.

• Documentation and stakeholder review.

• Infrastructure upgrades.

These measurements can replace our hypothetical maintenance estimates with organization-specific evidence.

Without them, maintenance comparisons remain informed estimates rather than empirical results.

14. Final verdict: make complexity earn its place

Our experiment produced three important findings.

First, the learned models did improve predictive performance. Boosting was the accuracy leader, although the difference from the compact RuleFit variant was tiny.

Second, implementation choices mattered as much as model family. The hand rules were dramatically faster in this benchmark, while our RuleFit variant incurred more inference overhead than histogram boosting.

Third, a measured accuracy win did not automatically determine the economic winner. Under higher missed-delivery costs, boosting offered substantial value. Under lower error costs and nontrivial maintenance expenses, the hand-written rules could be the more rational choice.

The most useful takeaway is not that rules are better, or that boosting is better.

It is that the incumbent baseline deserves to be treated as a serious competitor, and every additional layer of complexity should have to demonstrate its value.

A rules system is not an embarrassing temporary solution. A well-designed learned model is not unnecessary sophistication. Both are engineering choices that should be judged against the same operational objectives.

Your next step: run the bake-off on your own problem

Run the complete benchmark, then change the simulated warehouse conditions. Increase the frequency of unusual interactions, change the late-delivery costs, or increase the distribution shift.

Observe how the winner changes.

Then replace the synthetic dataset with a properly governed dataset from your own everyday business domain, preserving the chronological split and evaluation methodology.

Before selecting a production model, write down three numbers: the improvement in relevant errors, the monetary value of that improvement, and the estimated recurring cost of supporting the new system.

Make your next model selection decision with those numbers—not with accuracy alone.