Ship the rule-based baseline first: why 20 lines of if/else gate every good ML project

LEARN · RULES, HEURISTICS & SYMBOLIC AI

Why the First 20 Lines Matter

When people start an ML project, the instinct is often to train a model immediately. Open a notebook, split the data, fit a classifier, and compare metrics.

That feels productive—but it’s often backwards.

A better workflow is:

Build the simplest possible baseline.
Measure it.
Only then introduce machine learning.

Those first 20 lines of if/else logic can answer an expensive question early:

Does this problem actually need machine learning?

If the answer is “not really,” you’ve just saved days or weeks of engineering effort.

This article walks through three increasingly capable approaches:

A majority-class baseline
A small RuleBased heuristic
A trained Logistic Regression model

We’ll compare all three on the same dataset using scikit-learn 1.7 (the current stable release in 2025), discuss the honest performance delta, and explain why experienced ML teams almost always start here.

What Is a Baseline?

A baseline is the simplest reasonable solution to a prediction problem.

Good Baselines are:

Fast to implement
Easy to understand
Easy to debug
Cheap to maintain
Difficult to accidentally overfit

Their job is not to be impressive.

Their job is to establish the minimum level of performance that every future model must beat.

If your carefully tuned ML model barely outperforms a handful of deterministic rules, you should seriously ask whether the added complexity is worthwhile.

Three Levels of Increasing Complexity

Think of predictive systems as a ladder.

Level	Approach	Training Required
1	Majority-class predictor	No
2	RuleBased heuristics	No
3	Machine learning	Yes

Each level should earn its place.

Level 1: Majority-Class Baseline

The simplest possible classifier predicts the most common class every time.

Imagine a spam dataset where:

90% of emails are legitimate
10% are spam

Predicting “not spam” for every email gives:

Accuracy: 90%
Precision for spam: 0%
Recall for spam: 0%

It would be a terrible product.

It is an excellent benchmark.

Using DummyClassifier makes this almost effortless.

from sklearn.datasets import load_iris
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.30,
    stratify=y,
    random_state=42,
)

baseline = DummyClassifier(strategy="most_frequent")
baseline.fit(X_train, y_train)

predictions = baseline.predict(X_test)

print(f"Accuracy: {accuracy_score(y_test, predictions):.3f}")

On the Iris dataset, classes are balanced, so this baseline typically achieves roughly 33% accuracy.

That number is intentionally low.

Every future model now has a benchmark to beat.

Level 2: A Tiny RuleBased Baseline

Next comes a deterministic heuristic.

No optimization.

No gradient descent.

No training.

Just a few rules based on domain knowledge.

For Iris, petal measurements separate species surprisingly well.

from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.30,
    stratify=y,
    random_state=42,
)

def predict_rule(features):
    sepal_length, sepal_width, petal_length, petal_width = features

    if petal_length < 2.5:
        return 0

    if petal_width > 1.7:
        return 2

    return 1

predictions = [predict_rule(row) for row in X_test]

print(f"Accuracy: {accuracy_score(y_test, predictions):.3f}")

This is pure Heuristics.

Nothing is learned from data.

Yet the result is often surprisingly competitive because the dataset has clear feature separation.

Many beginners underestimate how powerful a few carefully chosen thresholds can be.

Level 3: Train a Real Model

Now it’s finally time for machine learning.

We’ll use Logistic Regression because it’s:

Fast
Well understood
Interpretable
Often difficult to beat on structured tabular data

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.30,
    stratify=y,
    random_state=42,
)

model = LogisticRegression(max_iter=500, random_state=42)

model.fit(X_train, y_train)

predictions = model.predict(X_test)

print(f"Accuracy: {accuracy_score(y_test, predictions):.3f}")

Nothing fancy.

No neural network.

No hyperparameter search.

Just a strong classical baseline model.

An Honest Comparison

Running all three approaches with the same train/test split typically produces results close to these:

Approach	Typical Accuracy
Majority-class baseline	0.33
RuleBased heuristic	0.96
Logistic Regression	0.98

The exact values depend on the train/test split, but the pattern is remarkably consistent.

The important observation is not that Logistic Regression wins.

It’s how much it wins.

The improvement is roughly:

Majority → Rules: +63 percentage points
Rules → Logistic Regression: +2 percentage points

That second improvement is the honest delta.

Now ask yourself:

Is a 2-point gain worth introducing model training?
Is retraining infrastructure justified?
Is monitoring prediction drift necessary?
Will deployment become more complicated?
Does the business actually benefit?

Sometimes the answer is absolutely yes.

Sometimes the answer is surprisingly no.

Why Simple Heuristics Reveal So Much

Rule-based systems provide more than accuracy.

They expose feature quality

If a few thresholds already work well, your features likely contain strong signal.

If simple rules perform terribly, more sophisticated models may struggle too.

They reveal labeling problems

Imagine a fraud dataset where your rule says:

Transaction amount > 10,000
New account
Foreign payment

…yet the labels repeatedly say “safe.”

That inconsistency might indicate:

Incorrect labels
Missing business context
Data collection issues

A simple heuristic often uncovers dataset problems before any model does.

They improve conversations with domain experts

A business stakeholder understands:

If petal length < 2.5, predict Setosa.

Much more easily than:

The learned decision boundary separates feature vectors in four-dimensional space.

Simple explanations make validation dramatically easier.

They speed up iteration

Changing a rule takes seconds.

Retraining a model may take minutes, hours, or even days.

Fast feedback loops usually produce better products.

Don’t Judge Everything by Accuracy

Accuracy is useful—but it can be misleading.

Depending on your application, also measure:

Precision
Recall
F1-score
ROC AUC
Confusion matrix

For heavily imbalanced datasets, these metrics are often much more informative than raw accuracy.

A model with 99% accuracy can still fail catastrophically if it misses every positive case.

Common Beginner Mistakes

Training without a baseline

If your first model reaches 94% accuracy, is that good?

You can’t know.

Maybe a dummy classifier already reaches 93%.

Without a baseline, every result lacks context.

Building an overly clever baseline

The goal isn’t to “beat” machine learning.

The goal is to establish a fair reference point.

If your baseline grows into hundreds of lines of logic, you’ve built another production system instead of a benchmark.

Ignoring operational costs

A deployed ML model isn’t just Python code.

It also requires:

Retraining
Monitoring
Versioning
Validation
Rollback plans
Drift detection
Observability

A simple rule file often requires none of these.

Operational simplicity has real business value.

Cherry on the Cake: The Product Story Everyone Eventually Encounters

One of the best-known real examples comes from Google’s original web search.

In the early days of Google Search, ranking was never “just one ML model.” Instead, it combined many deterministic signals and heuristics with learned components over time. Even today, search ranking remains a layered system where hard rules, safety checks, policy enforcement, and engineered signals work alongside machine learning.

Why?

Because deterministic logic excels at things like:

Blocking impossible states
Enforcing product policies
Applying safety constraints
Handling edge cases predictably
Reacting immediately during incidents

The lesson is surprisingly consistent across modern products.

The heuristic ships first.

The ML model, if it arrives, becomes another layer—not a replacement for everything that came before.

That’s why experienced engineering teams rarely frame the decision as “rules or ML.”

Instead, they ask:

Which parts should remain deterministic, and where does learning add measurable value?

A Practical Workflow You Can Reuse

A disciplined ML project often follows this sequence:

Understand the business problem.
Build a majority-class baseline.
Add a small RuleBased heuristic.
Measure both carefully.
Train a simple model such as Logistic Regression.
Compare the honest delta.
Introduce more complex models only if they provide meaningful additional value.

Skipping directly to deep learning may be exciting, but it often delays learning what the data could have told you in an afternoon.

Key Takeaways

Every ML project deserves simple Baselines before sophisticated models.
A majority-class predictor establishes the minimum acceptable benchmark.
RuleBased systems and Heuristics frequently perform far better than expected.
Always compare new models against your strongest simple baseline—not against zero.
Small accuracy gains should be weighed against operational complexity.
Explainability, maintainability, deployment cost, and debugging effort are part of model performance too.
Strong engineering teams optimize for business outcomes, not model sophistication.

The next time you start an ML project, resist the urge to train a model first. Write the 20-line rule-based baseline, measure it honestly, and let the numbers decide whether machine learning has truly earned its place.