Encoders shootout: one-hot, ordinal, target, and hashing — when each wins and when it leaks

Why categorical encoding deserves a shootout

Categorical features are everywhere in tabular machine learning:

• Country

• Browser

• Occupation

• Product ID

• Merchant

• ZIP code

Yet most machine learning algorithms expect numeric inputs. Your encoding strategy directly affects:

• Model accuracy

• Training time

• Memory consumption

• Generalization to unseen categories

• Risk of target leakage

There is no universally best encoder.

Instead, the winning approach depends on:

• Feature cardinality

• Model family

• Dataset size

• Whether unseen categories appear in production

• Whether the encoder learns from the target

In this hands-on guide, you’ll compare four common approaches using the same dataset, the same cross-validation strategy, and the same evaluation metric:

• OneHotEncoder

• OrdinalEncoder

• TargetEncoder

• HashingVectorizer

You’ll also reproduce one of the easiest mistakes in Feature Engineering—target leakage—and fix it correctly with a cross-validation-safe pipeline.

---

Environment

The examples below were tested with current library releases (mid-2025+ ecosystem):

• Python 3.12+

• scikit-learn 1.7.x

• category_encoders 2.8.x

• pandas 2.2+

• numpy 2.x

Install everything:

pip install -U scikit-learn category_encoders pandas numpy scipy

---

The dataset

We’ll use the Adult Income dataset from OpenML.

It predicts whether annual income exceeds $50K based on demographic and employment information.

It is ideal because it contains:

• Numeric features

• Low-cardinality categorical variables

• Medium-cardinality categorical variables

• Enough rows for meaningful cross-validation

import pandas as pd

from sklearn.datasets import fetch_openml

adult = fetch_openml(
    "adult",
    version=2,
    as_frame=True,
)

X = adult.data
y = (adult.target == ">50K").astype(int)

categorical = X.select_dtypes(include=["category", "object"]).columns.tolist()
numeric = X.select_dtypes(exclude=["category", "object"]).columns.tolist()

print(X.shape)
print(len(categorical), "categorical columns")

---

Common preprocessing

Every experiment will use identical preprocessing for numeric features.

from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

numeric_pipeline = Pipeline(
    [
        ("imputer", SimpleImputer(strategy="median")),
    ]
)

Using identical preprocessing ensures that differences come from the encoder—not from inconsistent data preparation.

---

1. OneHotEncoder

One-hot encoding creates one binary feature per category.

Example:

Color	Red	Blue	Green
Red	1	0	0
Blue	0	1	0

Advantages:

• No artificial ordering

• Excellent for linear models

• Highly interpretable

• Safe default for low-cardinality features

Disadvantages:

• Sparse matrices

• Feature explosion

• High memory usage with many categories

from sklearn.compose import ColumnTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder

cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

onehot = ColumnTransformer(
    [
        ("num", numeric_pipeline, numeric),
        (
            "cat",
            OneHotEncoder(handle_unknown="ignore"),
            categorical,
        ),
    ]
)

model = Pipeline(
    [
        ("prep", onehot),
        ("clf", LogisticRegression(max_iter=3000)),
    ]
)

scores = cross_val_score(
    model,
    X,
    y,
    cv=cv,
    scoring="accuracy",
)

print(scores.mean())

Typical accuracy is around 0.85.

---

2. OrdinalEncoder

Ordinal encoding replaces each category with an integer.

Example:

Color	Encoded
Red	0
Blue	1
Green	2

Although these numbers introduce an arbitrary order, modern tree-based models usually split on thresholds rather than treating the values as continuous quantities.

This makes ordinal encoding surprisingly competitive for gradient-boosted trees.

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.preprocessing import OrdinalEncoder

ordinal = ColumnTransformer(
    [
        ("num", numeric_pipeline, numeric),
        (
            "cat",
            OrdinalEncoder(
                handle_unknown="use_encoded_value",
                unknown_value=-1,
            ),
            categorical,
        ),
    ]
)

model = Pipeline(
    [
        ("prep", ordinal),
        ("clf", HistGradientBoostingClassifier(random_state=42)),
    ]
)

scores = cross_val_score(
    model,
    X,
    y,
    cv=cv,
    scoring="accuracy",
)

print(scores.mean())

Typical accuracy is around 0.87.

For linear models, however, this artificial ordering often hurts performance.

---

3. TargetEncoder

Target encoding replaces every category with the average target value observed during training.

Example:

City	Encoded value
London	0.74
Paris	0.53
Tokyo	0.62

Instead of creating hundreds or thousands of dummy variables, every category becomes a single informative feature.

This often works especially well for high-cardinality categorical variables.

from category_encoders import TargetEncoder

target = ColumnTransformer(
    [
        ("num", numeric_pipeline, numeric),
        (
            "cat",
            TargetEncoder(),
            categorical,
        ),
    ]
)

model = Pipeline(
    [
        ("prep", target),
        ("clf", HistGradientBoostingClassifier(random_state=42)),
    ]
)

scores = cross_val_score(
    model,
    X,
    y,
    cv=cv,
    scoring="accuracy",
)

print(scores.mean())

Typical accuracy is around 0.88.

---

The target leakage trap

Target encoding is powerful precisely because it learns from the labels.

That also makes it easy to misuse.

A common mistake is fitting the encoder before cross-validation.

from category_encoders import TargetEncoder
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

preprocessor = ColumnTransformer(
    [
        ("num", SimpleImputer(strategy="median"), numeric),
        ("cat", TargetEncoder(), categorical),
    ]
)

# ❌ Leakage: preprocessing sees the entire dataset first
X_encoded = preprocessor.fit_transform(X, y)

model = HistGradientBoostingClassifier(random_state=42)

scores = cross_val_score(
    model,
    X_encoded,
    y,
    cv=cv,
    scoring="accuracy",
)

print(scores.mean())

The encoder has already seen every target value—including those belonging to future validation folds.

Validation accuracy becomes overly optimistic.

The correct solution

Always put the encoder inside the pipeline.

safe_model = Pipeline(
    [
        ("prep", target),
        ("clf", HistGradientBoostingClassifier(random_state=42)),
    ]
)

scores = cross_val_score(
    safe_model,
    X,
    y,
    cv=cv,
    scoring="accuracy",
)

print(scores.mean())

Now every fold computes target statistics using only its own training partition.

This is the correct experimental protocol.

Whenever an encoder learns from y, the encoder must be fitted separately inside every training fold.

---

4. HashingVectorizer for categorical features

Unlike one-hot encoding, hashing never stores a category dictionary.

Instead, every category string is hashed into a fixed number of feature buckets.

Although FeatureHasher is often shown for structured data, HashingVectorizer provides an equally practical and fully stateless alternative by treating each row as a tiny “document” of feature tokens.

from scipy.sparse import hstack
from sklearn.feature_extraction.text import HashingVectorizer

docs = (
    X[categorical]
    .fillna("missing")
    .astype(str)
    .apply(
        lambda row: " ".join(
            f"{col}={row[col]}"
            for col in categorical
        ),
        axis=1,
    )
)

hasher = HashingVectorizer(
    n_features=2**12,
    alternate_sign=False,
    norm=None,
)

X_hash = hasher.transform(docs)

X_num = numeric_pipeline.fit_transform(X[numeric])

X_all = hstack([X_num, X_hash])

model = LogisticRegression(max_iter=3000)

scores = cross_val_score(
    model,
    X_all,
    y,
    cv=cv,
    scoring="accuracy",
)

print(scores.mean())

Typical accuracy usually falls between 0.84 and 0.87, depending on the number of hash buckets.

Advantages:

• Constant memory usage

• Naturally handles unseen categories

• No lookup tables

• Excellent for streaming data

Trade-offs:

• Hash collisions

• Less interpretability

• Performance depends on bucket count

---

Side-by-side comparison

Encoder	Typical accuracy	Best with	Main drawback
OneHotEncoder	~0.85	Linear models	Wide sparse matrices
OrdinalEncoder	~0.87	Tree ensembles	Artificial ordering
TargetEncoder	~0.88	High-cardinality features	Leakage risk
HashingVectorizer	~0.84–0.87	Huge vocabularies	Hash collisions

The exact numbers will vary by random seed, preprocessing, and library versions, but running every model with the same cross-validation pipeline makes the comparison fair.

---

When each encoder wins

Choose OneHotEncoder when

• Categories are relatively few

• Interpretability matters

• You’re using logistic regression or linear SVMs

• Memory is not a bottleneck

Choose OrdinalEncoder when

• Training gradient-boosted trees

• Categories are mostly nominal

• You want a lightweight preprocessing pipeline

Choose TargetEncoder when

• Features have hundreds or thousands of unique values

• Categories contain predictive signal

• You have enough observations per category

• The encoder is safely inside cross-validation

Choose HashingVectorizer when

• IDs continually grow

• User names or URLs appear

• Dictionaries become too large

• Online or streaming systems are involved

---

Cherry on the cake: a real Kaggle leakage lesson

One of Kaggle’s best-known educational examples of target leakage comes from the Intermediate Machine Learning course’s lesson on leakage. The course demonstrates how seemingly harmless preprocessing can allow information from the validation set to influence training, producing unrealistically strong validation scores. The lesson has become a standard reference precisely because similar mistakes repeatedly appeared in competition notebooks, especially when target-dependent features or encoders were computed before cross-validation.

Target encoding is one of the easiest places to introduce exactly this bug. If category averages are computed using the full dataset, every validation fold indirectly contributes to its own encoded features. Local cross-validation scores can look excellent, only to disappoint on the private leaderboard or in production.

The takeaway is simple: if a preprocessing step learns from the target, it belongs inside a cross-validation pipeline—not before it.

---

Key takeaways

• OneHotEncoding remains the safest default for low-cardinality categorical variables.

• OrdinalEncoder often performs well with modern tree-based models despite assigning arbitrary integers.

• TargetEncoding frequently wins on high-cardinality data, but only when fitted separately inside each training fold.

• HashingVectorizer scales gracefully to massive categorical vocabularies without maintaining lookup tables.

• In ScikitLearn, pipelines are your strongest defense against data leakage.

• Good FeatureEngineering is not just about extracting more signal—it is about preserving a valid evaluation protocol.

Choosing the right encoder usually matters less than evaluating it correctly.

A leakage-free experiment with slightly lower validation accuracy is far more valuable than an impressive score that cannot be reproduced.

---

Call to action

Take one of your existing tabular machine learning projects and benchmark all four encoders—OneHotEncoder, OrdinalEncoder, TargetEncoder, and HashingVectorizer—using identical cross-validation splits, identical metrics, and pipeline-based preprocessing. Record not only accuracy, but also training time, feature dimensionality, and memory usage. The results may surprise you, and you’ll build a reusable evaluation workflow that is far more valuable than any single encoding technique.