RuleFit and skope-rules: mining crisp rules out of a trained ensemble

Tree ensembles are excellent at discovering nonlinear thresholds and feature interactions. A boosted classifier can learn that a large order is suspicious only when the account is young, or that a distant transaction matters primarily when it comes from a new device.

The problem appears after training.

A probability such as 0.87 is easy for software to consume, but difficult to translate into:

• A policy an operations team can approve.

• A SQL filter an analyst can inspect.

• A reason code an investigator can understand.

• A compact set of branches engineers can test exhaustively.

• A monitoring rule whose activation rate can be tracked over time.

Rule mining offers a practical compromise. Let tree-based learners search a complicated feature space, then compress what they discover into conditions such as:

amount > 214.7 and account_age_days <= 31.5

failed_logins_24h > 1.5 and device_velocity_24h > 4.5

is_new_device > 0.5 and distance_km > 73.2

These conditions are more than annotations attached to another predictor. After selection and calibration, they can become the predictor.

In this tutorial, you will:

• Build a neutral synthetic e-commerce transaction dataset.

• Train a gradient-boosting reference model.

• Train rules-only RuleFit and Skope-rules classifiers.

• Inspect their extracted conditions.

• Retain only five rules from each method.

• Calibrate the compressed models without touching the test set.

• Measure how much accuracy, balanced accuracy, precision, recall, F1, and ROC AUC survive.

• Turn the result into something that can plausibly become production policy.

The current imodels setup

As of August 4, 2026, PyPI lists imodels 3.0.0, released on August 3, 2026. The package requires Python 3.9 or newer. Its rule-based estimators expose a common get_rules() interface that returns a pandas table containing at least the human-readable rule and its prediction value; model-specific implementations can add fields such as coefficient, support, type, and importance.

That unified interface is useful, but the two models in this tutorial should not be treated as interchangeable. They use tree paths differently and optimize different objectives.

RuleFit: sparse additive rule selection

A rules-only RuleFit classifier has the conceptual form:

score(x) = intercept + coefficient₁ × rule₁(x) + ... + coefficientₖ × ruleₖ(x)

Each rule function returns either zero or one:

ruleⱼ(x) = 1 when the condition is true, otherwise 0

The current implementation generates candidate rules from paths through a supported tree ensemble, removes duplicate conditions, converts the rules into binary features, and fits a sparse linear model over those indicators. For binary classification, the sparse stage uses logistic regression with L1 regularization. Setting include_linear=False removes the original numeric features so that the fitted model is made from rules rather than a mixture of rules and linear terms.

RuleFit answers a question like:

Which weighted combination of conditions gives the strongest overall classifier?

A positive coefficient raises the positive-class score when its rule fires. A negative coefficient lowers it. Multiple rules can fire for the same row, allowing evidence to reinforce or offset other evidence.

An important implementation detail

Supplying tree_generator does not tell RuleFit to dissect an arbitrary, already-fitted production model.

The estimator clones the supplied generator and trains the clone while producing candidate rules. A previous fit on the object is not reused. The current extraction path supports selected scikit-learn gradient-boosting and random-forest estimators, along with supported XGBoost estimators.

That distinction matters. RuleFit is best understood as an interpretable model trained with an ensemble-based rule generator, not as a universal post-hoc parser for every frozen black box.

Skope-rules: high-value positive regions

Skope-rules takes a different approach.

The current classifier trains bagged shallow regression trees, extracts candidate paths, and evaluates those candidates on out-of-bag observations. Rules that fail configured minimum precision or recall thresholds are removed. Near-duplicates can then be merged, with the retained candidates ordered using rule-level precision and recall, including an F1-based ranking during pruning.

Skope-rules answers a question like:

Which individual regions identify the positive class with useful precision and recall?

That framing is particularly natural when the positive decision means:

• Send an order to manual review.

• Trigger an operational alert.

• Route an event to a specialist queue.

• Block or defer a high-risk action.

• Request additional verification.

A rule can cover only a small portion of all positives and still be valuable if the rows it catches are highly concentrated with positive outcomes.

The full classifier combines firing rules using their learned rule quality. Its current probability calculation weights activated rules using their stored out-of-bag precision values.

Both classifiers in this tutorial are binary classifiers. A multiclass task needs a decomposition such as one-versus-rest, or a different estimator designed for multiclass prediction.

Create a reproducible environment

Create a project directory and virtual environment:

mkdir rule-mining-demo
cd rule-mining-demo
python -m venv .venv
source .venv/bin/activate

Upgrade pip and install the package version used here:

python -m pip install --upgrade pip
python -m pip install "imodels==3.0.0"

Create the experiment file:

touch rule_mining.py

Run the finished program with:

python rule_mining.py

Design the experiment before training

The example uses synthetic online orders with a noisy needs_review target. It avoids personal, patient, and medical data while still providing realistic nonlinear structure.

The hidden data-generating process contains patterns such as:

• A large purchase from a very young account.

• A distant transaction made from a new device.

• Multiple failed logins combined with high device velocity.

• A shipping mismatch during an overnight, high-value purchase.

• A large coupon used by a brand-new account.

No single feature perfectly determines the target. Randomness is deliberately added so that the task remains probabilistic rather than becoming a trivial reconstruction exercise.

We will use three partitions:

• Training set: fits the reference model, RuleFit, Skope-rules, and the five-rule RuleFit refit.

• Validation set: chooses the compact RuleFit decision threshold and the compact Skope-rules quorum.

• Test set: measures final performance once, after every modeling choice is fixed.

The resulting proportions are 60% training, 20% validation, and 20% test.

That separation is essential. Choosing five rules is already a model-selection decision. Choosing a threshold or deciding whether one, two, or three rules must fire is another. None of those choices should inspect final test performance.

The complete runnable experiment

Place the following program in rule_mining.py:

from __future__ import annotations

import numpy as np
import pandas as pd
from imodels import RuleFitClassifier, SkopeRulesClassifier
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score,
    balanced_accuracy_score,
    f1_score,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import train_test_split

RANDOM_STATE = 42
N_TOP_RULES = 5


def make_transactions(
    n_rows: int = 18_000,
    random_state: int = RANDOM_STATE,
) -> tuple[pd.DataFrame, pd.Series]:
    """Create a neutral synthetic e-commerce review dataset."""
    rng = np.random.default_rng(random_state)

    frame = pd.DataFrame(
        {
            "amount": rng.lognormal(
                mean=3.8,
                sigma=0.85,
                size=n_rows,
            ),
            "account_age_days": rng.gamma(
                shape=2.2,
                scale=170.0,
                size=n_rows,
            ),
            "distance_km": rng.exponential(
                scale=22.0,
                size=n_rows,
            ),
            "device_velocity_24h": rng.poisson(
                lam=2.8,
                size=n_rows,
            ),
            "failed_logins_24h": rng.poisson(
                lam=0.45,
                size=n_rows,
            ),
            "coupon_pct": rng.choice(
                [0, 5, 10, 15, 20, 30, 40, 50],
                size=n_rows,
                p=[
                    0.30,
                    0.07,
                    0.15,
                    0.10,
                    0.15,
                    0.10,
                    0.08,
                    0.05,
                ],
            ),
            "hour": rng.integers(
                0,
                24,
                size=n_rows,
            ),
            "is_new_device": rng.binomial(
                1,
                0.20,
                size=n_rows,
            ),
            "shipping_mismatch": rng.binomial(
                1,
                0.12,
                size=n_rows,
            ),
            "items_in_cart": (
                rng.poisson(lam=3.5, size=n_rows) + 1
            ),
        }
    )

    overnight = (
        (frame["hour"] <= 5)
        | (frame["hour"] >= 23)
    )

    young_account_large_order = (
        (frame["amount"] > 220.0)
        & (frame["account_age_days"] < 35.0)
    )

    distant_new_device = (
        (frame["distance_km"] > 75.0)
        & (frame["is_new_device"] == 1)
    )

    login_velocity_pattern = (
        (frame["failed_logins_24h"] >= 2)
        & (frame["device_velocity_24h"] >= 5)
    )

    shipping_pattern = (
        (frame["shipping_mismatch"] == 1)
        & (frame["amount"] > 150.0)
        & overnight
    )

    coupon_pattern = (
        (frame["coupon_pct"] >= 40)
        & (frame["account_age_days"] < 14.0)
    )

    log_odds = (
        -2.4
        + 2.3 * young_account_large_order
        + 2.0 * distant_new_device
        + 2.5 * login_velocity_pattern
        + 1.8 * shipping_pattern
        + 1.6 * coupon_pattern
        + 0.35 * (frame["amount"] > 350.0)
        + 0.25 * (
            frame["device_velocity_24h"] >= 7
        )
    )

    review_probability = (
        1.0 / (1.0 + np.exp(-log_odds))
    )

    target = pd.Series(
        rng.binomial(
            1,
            review_probability,
        ),
        index=frame.index,
        name="needs_review",
        dtype="int64",
    )

    return frame, target


def make_rule_matrix(
    frame: pd.DataFrame,
    rules: list[str],
) -> np.ndarray:
    """Evaluate readable rule strings as binary columns."""
    columns: list[np.ndarray] = []

    for rule in rules:
        mask = frame.eval(
            rule,
            engine="python",
        )
        columns.append(
            mask.to_numpy(dtype=np.float64)
        )

    if not columns:
        return np.empty(
            (len(frame), 0),
            dtype=np.float64,
        )

    return np.column_stack(columns)


def choose_balanced_accuracy_threshold(
    scores: np.ndarray,
    target: pd.Series,
) -> float:
    """Choose a threshold using validation data only."""
    unique_scores = np.unique(scores)

    if len(unique_scores) == 1:
        return float(unique_scores[0])

    midpoint_thresholds = (
        unique_scores[:-1]
        + unique_scores[1:]
    ) / 2.0

    candidates = np.concatenate(
        [
            [
                np.nextafter(
                    unique_scores[0],
                    -np.inf,
                )
            ],
            midpoint_thresholds,
            [
                np.nextafter(
                    unique_scores[-1],
                    np.inf,
                )
            ],
        ]
    )

    best_threshold = float(candidates[0])
    best_score = -np.inf

    for threshold in candidates:
        prediction = (
            scores >= threshold
        ).astype(int)

        score = balanced_accuracy_score(
            target,
            prediction,
        )

        if score > best_score:
            best_score = score
            best_threshold = float(threshold)

    return best_threshold


def choose_quorum(
    votes: np.ndarray,
    target: pd.Series,
    max_rules: int,
) -> int:
    """Choose how many selected Skope rules must fire."""
    best_quorum = 1
    best_score = -np.inf

    for quorum in range(1, max_rules + 1):
        prediction = (
            votes >= quorum
        ).astype(int)

        score = balanced_accuracy_score(
            target,
            prediction,
        )

        if score > best_score:
            best_score = score
            best_quorum = quorum

    return best_quorum


def metric_row(
    name: str,
    target: pd.Series,
    prediction: np.ndarray,
    score: np.ndarray | None = None,
) -> dict[str, float | str]:
    row: dict[str, float | str] = {
        "model": name,
        "accuracy": accuracy_score(
            target,
            prediction,
        ),
        "balanced_accuracy": (
            balanced_accuracy_score(
                target,
                prediction,
            )
        ),
        "precision": precision_score(
            target,
            prediction,
            zero_division=0,
        ),
        "recall": recall_score(
            target,
            prediction,
            zero_division=0,
        ),
        "f1": f1_score(
            target,
            prediction,
            zero_division=0,
        ),
        "roc_auc": np.nan,
    }

    if score is not None:
        row["roc_auc"] = roc_auc_score(
            target,
            score,
        )

    return row


def main() -> None:
    X, y = make_transactions()
    feature_names = X.columns.tolist()

    X_build, X_test, y_build, y_test = (
        train_test_split(
            X,
            y,
            test_size=0.20,
            stratify=y,
            random_state=RANDOM_STATE,
        )
    )

    X_train, X_valid, y_train, y_valid = (
        train_test_split(
            X_build,
            y_build,
            test_size=0.25,
            stratify=y_build,
            random_state=RANDOM_STATE,
        )
    )

    print(
        f"Train rows:      {len(X_train):,}"
    )
    print(
        f"Validation rows: {len(X_valid):,}"
    )
    print(
        f"Test rows:       {len(X_test):,}"
    )
    print(
        f"Positive rate:   {y.mean():.3%}"
    )

    reference = GradientBoostingClassifier(
        n_estimators=300,
        learning_rate=0.05,
        max_leaf_nodes=4,
        subsample=0.80,
        random_state=RANDOM_STATE,
    )

    reference.fit(
        X_train,
        y_train,
    )

    reference_test_prediction = (
        reference.predict(X_test)
    )

    reference_test_probability = (
        reference.predict_proba(X_test)[:, 1]
    )

    rulefit = RuleFitClassifier(
        n_estimators=300,
        tree_size=4,
        max_rules=40,
        tree_generator=(
            GradientBoostingClassifier(
                learning_rate=0.05,
                subsample=0.80,
            )
        ),
        include_linear=False,
        cv=True,
        random_state=RANDOM_STATE,
    )

    rulefit.fit(
        X_train.to_numpy(
            dtype=np.float64,
        ),
        y_train.to_numpy(
            dtype=int,
        ),
        feature_names=feature_names,
    )

    rulefit_test_prediction = (
        rulefit.predict(
            X_test.to_numpy(
                dtype=np.float64,
            )
        )
    )

    rulefit_test_probability = (
        rulefit.predict_proba(
            X_test.to_numpy(
                dtype=np.float64,
            )
        )[:, 1]
    )

    rulefit_rules = rulefit.get_rules()

    rulefit_candidates = (
        rulefit_rules.loc[
            (
                rulefit_rules["type"]
                == "rule"
            )
            & (
                rulefit_rules["coef"].abs()
                > 1e-12
            )
        ]
        .copy()
    )

    top_rulefit = (
        rulefit_candidates
        .sort_values(
            [
                "importance",
                "support",
            ],
            ascending=[
                False,
                False,
            ],
        )
        .head(N_TOP_RULES)
        .reset_index(drop=True)
    )

    if len(top_rulefit) < N_TOP_RULES:
        raise RuntimeError(
            "Fewer than five RuleFit rules "
            "survived. Increase max_rules "
            "or n_estimators."
        )

    rulefit_rule_strings = (
        top_rulefit["rule"].tolist()
    )

    rulefit_train_matrix = (
        make_rule_matrix(
            X_train,
            rulefit_rule_strings,
        )
    )

    rulefit_valid_matrix = (
        make_rule_matrix(
            X_valid,
            rulefit_rule_strings,
        )
    )

    rulefit_test_matrix = (
        make_rule_matrix(
            X_test,
            rulefit_rule_strings,
        )
    )

    rulefit_top5_model = (
        LogisticRegression(
            penalty="l2",
            C=1.0,
            solver="lbfgs",
            max_iter=2_000,
            random_state=RANDOM_STATE,
        )
    )

    rulefit_top5_model.fit(
        rulefit_train_matrix,
        y_train,
    )

    rulefit_top5_valid_probability = (
        rulefit_top5_model.predict_proba(
            rulefit_valid_matrix
        )[:, 1]
    )

    rulefit_top5_threshold = (
        choose_balanced_accuracy_threshold(
            rulefit_top5_valid_probability,
            y_valid,
        )
    )

    rulefit_top5_test_probability = (
        rulefit_top5_model.predict_proba(
            rulefit_test_matrix
        )[:, 1]
    )

    rulefit_top5_test_prediction = (
        rulefit_top5_test_probability
        >= rulefit_top5_threshold
    ).astype(int)

    rulefit_deployment_table = (
        top_rulefit[
            [
                "rule",
                "coef",
                "support",
                "importance",
            ]
        ]
        .copy()
    )

    rulefit_deployment_table[
        "refit_coef"
    ] = rulefit_top5_model.coef_[0]

    print(
        "\nTop five RuleFit rules:"
    )

    print(
        rulefit_deployment_table
        .to_string(index=False)
    )

    print(
        "RuleFit top-five validation "
        "threshold: "
        f"{rulefit_top5_threshold:.4f}"
    )

    skope = SkopeRulesClassifier(
        precision_min=0.20,
        recall_min=0.01,
        n_estimators=60,
        max_samples=0.80,
        max_samples_features=1.0,
        bootstrap=False,
        bootstrap_features=False,
        max_depth=[2, 3, 4],
        max_depth_duplication=2,
        max_features=1.0,
        min_samples_split=20,
        n_jobs=-1,
        random_state=RANDOM_STATE,
    )

    skope.fit(
        X_train.to_numpy(
            dtype=np.float64,
        ),
        y_train.to_numpy(
            dtype=int,
        ),
        feature_names=feature_names,
    )

    skope_test_prediction = (
        skope.predict(
            X_test.to_numpy(
                dtype=np.float64,
            )
        )
    )

    skope_test_probability = (
        skope.predict_proba(
            X_test.to_numpy(
                dtype=np.float64,
            )
        )[:, 1]
    )

    skope_public_rules = (
        skope.get_rules()
    )

    print(
        "\nPublic Skope rule table sample:"
    )

    print(
        skope_public_rules
        .head()
        .to_string(index=False)
    )

    skope_rule_table = pd.DataFrame(
        [
            {
                "rule": str(rule),
                "precision_oob": float(
                    rule.args[0]
                ),
                "recall_oob": float(
                    rule.args[1]
                ),
                "duplicate_count": int(
                    rule.args[2]
                ),
            }
            for rule in skope.rules_
        ]
    )

    if skope_rule_table.empty:
        raise RuntimeError(
            "Skope-rules produced no "
            "surviving rules. Lower "
            "precision_min or recall_min."
        )

    denominator = (
        skope_rule_table[
            "precision_oob"
        ]
        + skope_rule_table[
            "recall_oob"
        ]
    )

    skope_rule_table[
        "f1_oob"
    ] = np.where(
        denominator > 0.0,
        (
            2.0
            * skope_rule_table[
                "precision_oob"
            ]
            * skope_rule_table[
                "recall_oob"
            ]
            / denominator
        ),
        0.0,
    )

    skope_rule_table = (
        skope_rule_table
        .sort_values(
            [
                "f1_oob",
                "precision_oob",
                "recall_oob",
            ],
            ascending=[
                False,
                False,
                False,
            ],
        )
        .reset_index(drop=True)
    )

    top_skope = (
        skope_rule_table
        .head(N_TOP_RULES)
        .copy()
    )

    if len(top_skope) < N_TOP_RULES:
        raise RuntimeError(
            "Fewer than five Skope rules "
            "survived. Lower precision_min "
            "or recall_min."
        )

    skope_rule_strings = (
        top_skope["rule"].tolist()
    )

    skope_valid_matrix = (
        make_rule_matrix(
            X_valid,
            skope_rule_strings,
        )
    )

    skope_test_matrix = (
        make_rule_matrix(
            X_test,
            skope_rule_strings,
        )
    )

    skope_valid_votes = (
        skope_valid_matrix.sum(axis=1)
    )

    skope_test_votes = (
        skope_test_matrix.sum(axis=1)
    )

    skope_top5_quorum = choose_quorum(
        skope_valid_votes,
        y_valid,
        max_rules=N_TOP_RULES,
    )

    skope_top5_test_prediction = (
        skope_test_votes
        >= skope_top5_quorum
    ).astype(int)

    print(
        "\nTop five Skope rules:"
    )

    print(
        top_skope.to_string(index=False)
    )

    print(
        "Skope top-five validation "
        f"quorum: {skope_top5_quorum}"
    )

    results = pd.DataFrame(
        [
            metric_row(
                "Boosted reference",
                y_test,
                reference_test_prediction,
                reference_test_probability,
            ),
            metric_row(
                "RuleFit full",
                y_test,
                rulefit_test_prediction,
                rulefit_test_probability,
            ),
            metric_row(
                "RuleFit top 5",
                y_test,
                rulefit_top5_test_prediction,
                rulefit_top5_test_probability,
            ),
            metric_row(
                "Skope full",
                y_test,
                skope_test_prediction,
                skope_test_probability,
            ),
            metric_row(
                "Skope top 5",
                y_test,
                skope_top5_test_prediction,
                skope_test_votes,
            ),
        ]
    )

    reference_accuracy = float(
        results.loc[
            (
                results["model"]
                == "Boosted reference"
            ),
            "accuracy",
        ].iloc[0]
    )

    results[
        "accuracy_survival_pct"
    ] = (
        100.0
        * results["accuracy"]
        / reference_accuracy
    )

    print(
        "\nTest metrics:"
    )

    print(
        results
        .round(4)
        .to_string(index=False)
    )


if __name__ == "__main__":
    main()

Why the reference model is separate

The gradient-boosting classifier establishes a practical benchmark. It is the model against which compression is measured.

It is deliberately modest:

• 300 boosting stages.

• A low learning rate.

• At most four leaves per tree.

• Subsampling to introduce useful diversity.

The reference model is not passed into RuleFit after fitting. RuleFit receives a fresh generator specification and trains its own cloned ensemble while creating candidates.

This means the comparison is between three separately trained modeling strategies:

1. A conventional boosted reference.

2. A complete RuleFit classifier.

3. A complete Skope-rules classifier.

The five-rule variants are then compressed forms derived from the latter two models.

Why the RuleFit compression includes a refit

The full RuleFit model may retain many nonzero rules. Its coefficient for a given rule was learned in the presence of all those other rules.

Suppose the complete model contains these terms:

score = intercept
      + 1.2 × rule_a
      + 0.8 × rule_b
      - 0.6 × rule_c
      + 0.4 × rule_d
      + ...

Dropping everything except five terms while keeping the original coefficients creates a different score function, but the remaining weights were never optimized for that reduced feature set.

The tutorial therefore separates two operations:

1. Selection: use RuleFit’s importance ranking to choose five rules.

2. Calibration: evaluate those five conditions as binary columns and fit a new logistic regression using only those columns.

The refitted compact model contains:

• Five readable conditions.

• Five refitted coefficients.

• One intercept.

• One validation-selected decision threshold.

That is a genuine five-rule classifier rather than a partially amputated version of a larger scorecard.

The refit uses L2 regularization because the rule budget has already forced extreme sparsity. With only five columns, the goal is stable calibration rather than another selection pass.

How RuleFit ranks its candidates

For RuleFit, get_rules() includes fields such as:

• rule: the condition.

• type: whether the row is a rule or a linear term.

• coef: the sparse model coefficient.

• support: the fraction of training observations satisfying the rule.

• importance: a global importance score.

For a rule, the current global importance calculation is proportional to:

importance = |coefficient| × √(support × (1 − support))

That support adjustment is useful.

A rule that fires for almost every row adds little segmentation. A rule that fires for only a microscopic fraction may have a dramatic coefficient but be unstable. The importance expression gives more credit to rules that combine a meaningful coefficient with nondegenerate coverage.

The script sorts by importance and uses support as a tie-breaker. It also removes coefficients that are effectively zero before taking the first five candidates.

Why Skope-rules uses out-of-bag evaluation

Each Skope tree is trained on a sampled subset of the training observations. Rows omitted from that sample can be used to evaluate the tree’s extracted conditions.

This creates an inexpensive internal check:

• The tree proposes a region using its sampled rows.

• The rule is scored on rows that did not train that tree.

• Precision and recall thresholds remove weak candidates.

• Duplicate or near-duplicate rules are consolidated.

Setting max_samples=0.80 leaves observations available for that out-of-bag scoring. Using all rows without replacement would eliminate the natural holdout and cause the implementation to fall back to in-bag evaluation, which is less convincing for candidate screening.

The script asks for depths two, three, and four:

• Depth two produces short, broad conditions.

• Depth three captures moderate interactions.

• Depth four can isolate narrower regions.

The estimator trains the requested number of bagged trees for each supplied depth and pools their candidates.

Reading the Skope rule table

The portable get_rules() method provides a normalized public table. For deeper model-specific inspection, the fitted rules_ collection contains rule objects whose current scoring arguments hold:

• Out-of-bag precision.

• Out-of-bag recall.

• The duplicate count accumulated during pruning.

The script converts that fitted collection into a DataFrame and computes:

F1 = 2 × precision × recall / (precision + recall)

A high-precision, low-recall rule is a specialist.

For example, imagine a condition with:

precision = 0.82
recall = 0.04

It catches only 4% of all positive rows, but 82% of the rows it catches are positive. That may be excellent for a small, expensive investigation queue.

Another rule might have:

precision = 0.31
recall = 0.46

It is less concentrated, but covers far more of the target population. That rule behaves more like a generalist.

A useful operational policy often mixes both kinds.

Why the top-five Skope model tunes a quorum

The most natural compressed Skope policy is an OR:

positive when rule₁ or rule₂ or rule₃ or rule₄ or rule₅ fires

That corresponds to a quorum of one.

It is easy to explain, but it is not always the best decision boundary. Five overlapping rules may make a one-rule trigger too permissive. Requiring two or more conditions can improve precision, although recall will usually fall.

The script evaluates quorums from one through five on the validation set:

quorum = 1, 2, 3, 4, or 5

It selects the quorum with the highest validation balanced accuracy, then freezes that value before evaluating the test set.

For a real review system, the selection objective might instead be:

• Maximum recall while precision remains above 40%.

• Maximum precision while at least 20% of positives are covered.

• Lowest expected operational cost.

• A fixed daily review capacity.

• A business-specific utility function.

Balanced accuracy is a reasonable tutorial default because the target is imbalanced and both classes matter.

Understanding the metric table

The final output compares five predictors:

• Boosted reference.

• Full RuleFit.

• Five-rule RuleFit.

• Full Skope-rules.

• Five-rule Skope policy.

The reported metrics answer different questions.

Accuracy

Accuracy is the fraction of all decisions that are correct.

It remains useful, but can be misleading when positives are uncommon. A classifier that labels every order as normal may achieve high accuracy while catching no review-worthy cases.

Balanced accuracy

Balanced accuracy averages the true-positive rate and true-negative rate.

It gives each class equal influence even when one class is much larger, making it a better threshold-selection metric for this demonstration.

Precision

Precision asks:

Of the rows predicted positive, how many were actually positive?

High precision reduces wasted investigations.

Recall

Recall asks:

Of all actual positives, how many did the model catch?

High recall reduces missed cases.

F1

F1 combines precision and recall into a single harmonic mean.

It is useful when both matter, but it still assumes they deserve roughly symmetric treatment. Many production systems have asymmetric costs and should use an explicit business objective instead.

ROC AUC

ROC AUC evaluates the ranking quality of a continuous score across possible thresholds.

The full models use their predicted probabilities. The compact RuleFit model uses its refitted probability. The compact Skope model uses the number of selected rules firing; a higher vote count is treated as stronger positive evidence.

Measuring accuracy survival

The script adds this column:

accuracy survival (%) = compressed accuracy / boosted accuracy × 100

Suppose the boosted reference reaches 0.910 accuracy and a five-rule model reaches 0.891.

The survival calculation is:

0.891 / 0.910 × 100 = 97.91%

That does not mean the compressed model is “97.91% accurate.” Its actual accuracy is 89.1%.

It means the compact model retains approximately 97.91% of the reference model’s accuracy ratio.

This distinction is important because percentages can otherwise become misleading.

Never judge compression from that one column alone. A compact model can retain nearly all overall accuracy while losing much of the positive-class recall. Always read survival alongside:

• Balanced accuracy.

• Precision.

• Recall.

• F1.

• Activation volume.

• Rule coverage.

• Operational cost.

How to interpret an extracted condition

Suppose RuleFit returns:

account_age_days <= 33.8 and amount > 217.4

This identifies a region containing comparatively large orders from young accounts.

For RuleFit:

• A positive coefficient means the condition raises the positive score.

• A negative coefficient means the condition supplies evidence against the positive class.

• Support tells you how frequently it fired during training.

• Importance combines coefficient magnitude with coverage.

For Skope-rules:

• Precision estimates how concentrated the rule is with positives.

• Recall estimates how much of the positive population it captures.

• Duplicate count indicates how often similar candidates were consolidated.

A readable condition is not automatically a trustworthy policy. It still needs validation.

Very low support deserves investigation

A rare rule may represent:

• A genuinely important edge case.

• A new attack or abuse pattern.

• A small but costly operational segment.

• A threshold fitted to a handful of observations.

• A data-quality defect.

• Leakage from a downstream process.

• An unstable coincidence that will disappear next week.

Before approving it, inspect:

• The number of training rows that fire it.

• Validation and test precision.

• Performance across random seeds.

• Activation rate across time windows.

• Sensitivity to small threshold changes.

• Whether the underlying features are reliably available.

Support should be considered alongside business impact. A rule that fires only ten times per month can still be valuable if each event is extremely costly.

Validate rule stability

Rule mining converts subtle model behavior into explicit conditions. That improves auditability, but also exposes how unstable thresholds can be.

A stability experiment should repeat training across several seeds or bootstrap samples and track:

• How frequently each feature appears.

• Whether inequality directions stay consistent.

• How much numeric thresholds move.

• Support variation.

• Precision and recall variation.

• Pairwise overlap between selected rules.

• The number of times each operational pattern reappears.

A simple RuleFit stability loop can be added after the main experiment:

stability_rows: list[dict[str, object]] = []

for seed in [7, 21, 42, 84, 168]:
    model = RuleFitClassifier(
        n_estimators=300,
        tree_size=4,
        max_rules=40,
        tree_generator=GradientBoostingClassifier(
            learning_rate=0.05,
            subsample=0.80,
        ),
        include_linear=False,
        cv=True,
        random_state=seed,
    )

    model.fit(
        X_train.to_numpy(dtype=np.float64),
        y_train.to_numpy(dtype=int),
        feature_names=feature_names,
    )

    table = model.get_rules()

    selected = (
        table.loc[
            (table["type"] == "rule")
            & (table["coef"].abs() > 1e-12)
        ]
        .sort_values(
            "importance",
            ascending=False,
        )
        .head(5)
        .reset_index(drop=True)
    )

    for rank, row in selected.iterrows():
        stability_rows.append(
            {
                "seed": seed,
                "rank": rank + 1,
                "rule": row["rule"],
                "importance": float(
                    row["importance"]
                ),
                "support": float(
                    row["support"]
                ),
            }
        )

stability_table = pd.DataFrame(
    stability_rows
)

print(
    stability_table
    .sort_values(
        ["rank", "seed"]
    )
    .to_string(index=False)
)

Exact string matching is strict. These two rules are different strings:

distance_km > 74.8

distance_km > 76.1

Operationally, however, they may represent the same pattern.

A mature stability analysis should therefore compare more than raw text. Better options include:

• Grouping thresholds into business-approved bands.

• Parsing rules into normalized feature/operator structures.

• Measuring agreement between rule activation masks.

• Comparing the rows covered on a fixed validation sample.

• Clustering near-duplicate conditions before review.

Prefer time-based validation when production is temporal

Random train-validation-test splits are appropriate for a controlled tutorial. Many real systems should use chronological splits instead.

For example:

• Train on January through April.

• Validate on May.

• Test on June.

• Run a shadow deployment in July.

Time-based evaluation reveals whether rules depend on:

• Temporary promotions.

• Seasonal traffic.

• A short-lived abuse pattern.

• A recently introduced product flow.

• Changing customer behavior.

• Feature-definition changes.

A random split can hide these failures because observations from the same period appear in every partition.

Watch for information leakage

Rules are especially effective at exposing leakage because suspicious shortcuts become readable.

Imagine that an extracted condition says:

manual_review_completed > 0.5

That is not a valuable predictor. It is evidence that a feature created after the decision entered the training data.

Before approving a mined rule, ask:

• Was every feature available at scoring time?

• Could the value have changed after the outcome?

• Does the feature describe a downstream workflow?

• Is it a disguised timestamp or identifier?

• Is it populated only for known positives?

• Would the same field exist in a real-time request?

• Does an upstream join accidentally include future records?

Interpretability makes leakage easier to notice. It does not prevent leakage automatically.

Handle missing values deliberately

The extracted comparisons assume each referenced feature can be evaluated against a numeric threshold.

Do not allow every deployment runtime to invent its own missing-value behavior.

Choose an explicit policy:

• Impute values before model training and deploy the same imputation logic.

• Add missing-indicator features.

• Reject incomplete requests.

• Route incomplete rows to a fallback model.

• Send incomplete rows to manual review.

• Define a business-owned default branch.

If you impute, extract rules from values after imputation. Otherwise, the readable conditions may not describe what the model actually saw.

For example, suppose missing account ages are replaced with minus one. A rule such as:

account_age_days <= 12.5

then includes both genuinely young accounts and rows with missing account age. That may be acceptable, but the interpretation must say so.

Be careful with categorical features

One-hot encoded categories can produce readable conditions such as:

shipping_country_DE > 0.5

But large categorical vocabularies can generate:

• Many nearly identical rules.

• Rare-category thresholds with low support.

• Conditions tied to obsolete category levels.

• Operational policies that are difficult to maintain.

Before mining rules from categoricals:

• Consolidate rare levels.

• Freeze the vocabulary used in production.

• Define unknown-category behavior.

• Prefer business-meaningful groupings where possible.

• Verify that encoded column names are valid in your rule-evaluation environment.

The tutorial uses simple numeric and binary features so that every extracted expression can be evaluated directly with pandas.

Translate the compact model into production logic

The five-rule RuleFit refit is an additive scorecard:

score = intercept
score += coefficient₁ when rule₁ fires
score += coefficient₂ when rule₂ fires
score += coefficient₃ when rule₃ fires
score += coefficient₄ when rule₄ fires
score += coefficient₅ when rule₅ fires
probability = sigmoid(score)
prediction = probability >= threshold

The sigmoid function is:

sigmoid(score) = 1 / (1 + exp(−score))

The compact Skope policy is even simpler:

votes = number of selected rules that fire
prediction = votes >= quorum

Before deployment, export and version:

• The exact rule strings.

• The compact coefficients.

• The logistic intercept.

• The selected probability threshold.

• The Skope quorum.

• Feature definitions.

• Missing-value behavior.

• Package and model versions.

• Training-data dates.

• Validation metrics.

Do not rely on a notebook cell or console transcript as the authoritative policy definition.

Boundary semantics matter

Tree paths use exact comparison operators such as <= and >.

For every threshold, test:

• A value immediately below it.

• A value exactly equal to it.

• A value immediately above it.

• A missing value.

• Positive and negative infinity if the input layer permits them.

• An integer converted to a float.

• A string mistakenly supplied instead of a number.

• Values outside the physically possible range.

Changing this:

amount <= 214.7

into this:

amount < 214.7

changes the decision for exactly 214.7.

That boundary may rarely occur with continuous measurements, but it can occur frequently when values are rounded, bucketed, or represented in minor currency units.

Count more than rules

A five-rule model is not automatically simple.

Consider these two policies:

Five rules, each containing four conditions

Eight rules, each containing one condition

The eight-rule policy may be easier to understand, test, and monitor.

Measure complexity using:

• Number of rules.

• Total number of atomic comparisons.

• Maximum conjunction depth.

• Number of distinct features.

• Number of numeric thresholds.

• Overlap between rules.

• Number of exceptions.

• Number of feature pipelines involved.

• Number of teams that own dependencies.

Interpretability is a systems property, not merely a small integer printed as n_rules.

Monitor activation rates after deployment

A rule can remain syntactically valid while becoming operationally useless.

Track each condition’s:

• Daily or hourly activation count.

• Activation percentage.

• Positive outcome rate.

• Precision and recall when labels arrive.

• Overlap with other rules.

• Distribution by important segments.

• Missing-feature rate.

• Distance from its training support.

A sudden activation spike may indicate:

• A broken upstream feature.

• A unit conversion error.

• A product launch.

• A traffic-source change.

• Adversarial adaptation.

• A legitimate behavioral shift.

A rule that stops firing may indicate a renamed category, a changed default, or a pipeline that now emits values in different units.

Cherry on the cake: the 12-rule fraud handoff

Consider a realistic composite deployment story rather than a claimed named case study.

A fraud team operates a high-performing ensemble containing hundreds of trees. The model catches valuable cases, but the surrounding workflow has several problems:

• Analysts receive scores without stable reason codes.

• Every threshold adjustment requires a machine-learning deployment.

• Investigators cannot tell which behavioral pattern created an alert.

• Compliance reviewers see feature-attribution charts but no explicit decision policy.

• A broken feature can influence thousands of branches before anyone understands what changed.

The team mines several hundred candidate paths.

Sparse selection reduces them to roughly 30. Repeated training removes rules whose thresholds move dramatically between samples. Analysts reject conditions based on features unavailable at transaction time. Near-duplicates are merged into operational bands.

The final policy contains 12 rules.

Each rule receives:

• An owner.

• A plain-language description.

• A reason code.

• Expected daily volume.

• Validation precision and recall.

• A monitoring panel.

• A rollback switch.

• An expiration-review date.

The black-box ensemble remains online in shadow mode, but the 12-rule policy becomes the primary queueing mechanism.

The surprising result is not perfect imitation. Twelve rules do not reproduce hundreds of trees exactly.

The operational system improves despite a modest predictive loss:

• Investigators understand why alerts exist.

• Engineers can test every branch.

• Analysts can change approved thresholds without retraining.

• Monitoring identifies which condition drifted.

• Compliance discussions focus on explicit behavior.

• Rollbacks affect one policy version rather than an entire model stack.

This is the deeper purpose of rule mining.

The goal is not to prove that five or twelve rules are universally better than boosting. The goal is to find the smallest governed system that preserves enough predictive value for the decision being made.

When to choose RuleFit

RuleFit is a strong choice when:

• Rules should combine additively.

• Positive and negative evidence should offset each other.

• Overall predictive performance matters more than isolated rule precision.

• You want coefficient-based ranking.

• You need a compact score rather than a pure alert list.

• You want to retain linear terms in addition to rules for another project.

Its compact deployment resembles a conventional scorecard, except that the inputs are learned conditions rather than only raw variables.

When to choose Skope-rules

Skope-rules is a strong choice when:

• Individual high-value regions are the product.

• The positive class represents an alert or review queue.

• An OR or quorum policy is operationally natural.

• You want minimum precision and recall constraints.

• Out-of-bag rule diagnostics are useful.

• Analysts want to approve conditions individually.

Its rules can often be treated as modular detectors, each with its own owner, reason code, and monitoring threshold.

When to run both

Run both when the decision matters.

They may discover overlapping conditions but rank them differently because they solve different optimization problems.

A pattern can be:

• Highly useful inside RuleFit’s additive score.

• Too weak to survive Skope’s precision filter.

• Extremely precise in Skope-rules.

• Redundant once RuleFit includes a correlated negative rule.

Comparing the outputs can reveal which patterns are:

• Predictively useful.

• Individually actionable.

• Stable across methods.

• Easy to explain.

• Too dependent on interactions with other rules.

Agreement between the methods is not proof, but it is useful evidence during review.

A production-readiness checklist

Before replacing or supplementing an ensemble with mined rules, verify that:

• Rule selection used training or validation data, never the final test set.

• Thresholds and quorums were fixed before test evaluation.

• The compact model was evaluated on exactly the same untouched rows as the reference.

• Accuracy is accompanied by balanced accuracy, precision, recall, F1, and ranking metrics.

• All features exist at decision time.

• Missing-value behavior is explicit.

• Category vocabularies are versioned.

• Threshold boundaries have automated tests.

• Rule activation rates are monitored.

• Stability has been measured across seeds, resamples, or time windows.

• Rule overlap has been reviewed.

• Every rule has an owner.

• Every production policy has a rollback path.

• The full ensemble can run in shadow mode during rollout.

• Model versions and policy versions are recorded separately.

• Retraining cannot silently overwrite human-approved production logic.

• Rules have scheduled expiration or reapproval dates.

• A performance loss has been translated into business impact rather than reported only as a statistical percentage.

Your next step

Run the complete script and inspect the conditions selected on your machine.

Then make the compression harder:

1. Change N_TOP_RULES from five to three.

2. Rerun the experiment without changing the test set.

3. Compare accuracy survival, balanced accuracy, precision, recall, and F1.

4. Repeat with seven and ten rules.

5. Plot rule count against predictive performance and operational complexity.

6. Identify the smallest policy that still satisfies your real precision, recall, capacity, and governance requirements.

Do not stop after printing readable conditions.

Turn the selected rules into a versioned, boundary-tested, monitored decision policy—and record exactly how much of the ensemble’s performance survived the transformation.