Text features in Spark: Tokenizer, HashingTF, IDF, and Word2Vec on large corpora

Modern machine learning systems often start with a simple challenge: computers do not understand raw text. A support ticket, email, product review, or application log entry is just a sequence of characters. Machine learning algorithms need numbers, so engineers must transform text into useful numerical features.

Apache Spark’s spark.ml library provides scalable tools for this process. In current Spark 4.x environments, the DataFrame-based ML API continues to provide production-ready feature transformers such as:

• Tokenizer and RegexTokenizer for converting text into tokens

• HashingTF for scalable term-frequency features

• IDF for TF-IDF weighting

• Word2Vec for dense semantic embeddings

This workflow remains valuable for large-scale machine learning because it is fast, distributed, explainable, and easy to integrate into repeatable pipelines.

A common architecture looks like this:

Raw text
   ↓
Tokenizer
   ↓
Tokens
   ↓
HashingTF
   ↓
Term-frequency vectors
   ↓
IDF
   ↓
TF-IDF features
   ↓
Classifier, search, clustering, or analytics

The goal of this lesson is to build a practical understanding of Text features in Spark and learn when to use TFIDF-style features versus Word2Vec embeddings.

Why text features matter at scale

Processing a few thousand documents on a laptop is simple. Processing billions of text records requires distributed engineering.

Large organizations use Spark text pipelines for workloads such as:

• Spam detection across message streams

• Customer support ticket routing

• Security event classification

• Application log analytics

• Product catalog search

• Document similarity systems

A production text pipeline must handle:

• Distributed storage and computation

• Repeatable preprocessing

• Training and inference consistency

• Model persistence

• Changing language patterns over time

Spark ML pipelines solve part of this problem by packaging feature transformations and models into reusable stages.

Preparing text data with Spark DataFrames

Spark ML works naturally with DataFrames. A typical training dataset contains a label column and a text column.

Example:

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("SparkTextFeatures")
    .getOrCreate()
)

data = [
    (1, "Win a free prize now"),
    (0, "The project meeting is scheduled tomorrow"),
    (1, "Claim your reward by clicking this link"),
    (0, "Please review the attached report")
]

df = spark.createDataFrame(
    data,
    ["label", "text"]
)

df.show(truncate=False)

The challenge is converting the text column into a numerical representation that a machine learning model can consume.

Step 1: Tokenizer converts text into words

The first stage is usually tokenization.

For example:

"Spark makes big data processing easier"

becomes:

["spark", "makes", "big", "data", "processing", "easier"]

Spark provides both Tokenizer and RegexTokenizer. RegexTokenizer is often preferred in real applications because it gives more control over splitting rules.

Example:

from pyspark.ml.feature import RegexTokenizer

tokenizer = RegexTokenizer(
    inputCol="text",
    outputCol="words",
    pattern="\\W+"
)

tokenized = tokenizer.transform(df)

tokenized.select(
    "text",
    "words"
).show(truncate=False)

Tokenization is not just a mechanical step. The correct strategy depends on the domain.

For example:

• A sentiment model may remove punctuation and lowercase everything.

• A security log classifier may preserve identifiers such as ERR_CONNECTION_RESET.

• A product search system may need to keep model numbers and technical terms.

Poor tokenization can remove exactly the information a model needs.

Step 2: HashingTF creates scalable text vectors

Machine learning algorithms cannot directly use arrays of words. They require numerical features.

A traditional approach creates a vocabulary:

spark → 0
data → 1
model → 2

However, very large corpora may contain millions of unique terms. Storing and managing a complete vocabulary becomes expensive.

HashingTF uses the hashing trick:

word → hash function → vector position

Instead of storing every word, Spark maps terms into a fixed-size vector.

Example:

from pyspark.ml.feature import HashingTF

hashing_tf = HashingTF(
    inputCol="words",
    outputCol="rawFeatures",
    numFeatures=10000
)

tf_features = hashing_tf.transform(tokenized)

tf_features.select(
    "words",
    "rawFeatures"
).show(truncate=False)

The biggest advantage is predictable memory usage.

The tradeoff is collisions.

Two different words can theoretically map to the same location:

database → position 1532
security → position 1532

The model cannot perfectly separate those terms.

Increasing numFeatures reduces collisions but creates larger vectors:

• Smaller vectors: faster processing, more collisions

• Larger vectors: more memory, fewer collisions

For many large-scale classification tasks, this tradeoff is acceptable because the model learns statistical patterns from many examples.

Step 3: IDF adds information weighting

Term frequency alone is often not enough.

A word appearing thousands of times may not be useful if it appears in almost every document. Common words carry less information.

The IDF stage reduces the importance of widespread terms and increases the influence of more distinctive terms.

The idea behind TF-IDF is:

TF-IDF = term frequency × inverse document frequency

Spark’s IDF transformer learns these statistics from training data.

Example:

from pyspark.ml.feature import IDF

idf = IDF(
    inputCol="rawFeatures",
    outputCol="features"
)

idf_model = idf.fit(tf_features)

tfidf_features = idf_model.transform(tf_features)

tfidf_features.select(
    "text",
    "features"
).show(truncate=False)

The fit() step is important. The IDF model learns from the training corpus and must be reused consistently during prediction.

Building a complete Spark ML text classification pipeline

The strongest pattern is combining all stages into a single Spark ML pipeline.

This prevents common production mistakes, such as:

• Training with one preprocessing method

• Predicting with a different preprocessing method

• Forgetting to apply a transformation step

Example:

from pyspark.ml import Pipeline
from pyspark.ml.classification import LogisticRegression
from pyspark.ml.feature import RegexTokenizer, HashingTF, IDF

tokenizer = RegexTokenizer(
    inputCol="text",
    outputCol="words",
    pattern="\\W+"
)

hashing_tf = HashingTF(
    inputCol="words",
    outputCol="rawFeatures",
    numFeatures=10000
)

idf = IDF(
    inputCol="rawFeatures",
    outputCol="features"
)

classifier = LogisticRegression(
    featuresCol="features",
    labelCol="label"
)

pipeline = Pipeline(
    stages=[
        tokenizer,
        hashing_tf,
        idf,
        classifier
    ]
)

model = pipeline.fit(df)

predictions = model.transform(df)

predictions.select(
    "text",
    "prediction",
    "probability"
).show(truncate=False)

The same pipeline object can later be saved and loaded for production inference.

Word2Vec: moving from keywords to meaning

TF-IDF treats words as independent features.

For example, these words have separate positions:

king
queen
royal

A TF-IDF model does not naturally know that they are related.

Word2Vec creates dense vector representations where words used in similar contexts tend to have similar vectors.

Example:

from pyspark.ml.feature import Word2Vec

word2vec = Word2Vec(
    vectorSize=100,
    minCount=1,
    inputCol="words",
    outputCol="wordVectors"
)

word2vec_model = word2vec.fit(tokenized)

embeddings = word2vec_model.transform(tokenized)

embeddings.select(
    "text",
    "wordVectors"
).show(truncate=False)

Important Word2Vec parameters include:

• vectorSize: size of each embedding vector

• minCount: minimum number of occurrences required for a word

• Training corpus size: larger datasets generally produce better representations

Word2Vec is useful for:

• Document similarity

• Recommendation systems

• Search ranking

• Topic discovery

• Clustering

TF-IDF versus Word2Vec

Both approaches remain useful.

Feature method	Strengths	Limitations
TF-IDF	Fast, interpretable, strong baseline	Limited semantic understanding
HashingTF	Scales well, no vocabulary storage	Possible collisions
Word2Vec	Captures semantic relationships	Requires more data and training

A practical engineering approach is:

1. Start with HashingTF + IDF.

2. Measure baseline performance.

3. Add Word2Vec when semantic similarity matters.

4. Compare accuracy, latency, and operational cost.

The most complex model is not always the best production choice.

A Spark text pipeline success story: spam and log classification

One of the most practical uses of Spark text features is classifying large streams of messages and operational events.

A company processing millions of records per day may use a Spark ML pipeline to automatically:

• Identify spam messages

• Route support requests

• Group similar incidents

• Detect unusual application behavior

A TF-IDF classifier can be surprisingly effective when the domain vocabulary is highly informative.

For example, security logs often contain strong signals:

authentication failure
unknown device
repeated login
privilege escalation

A lightweight model can provide fast classification while remaining explainable to analysts.

This matters because many operational teams need answers to questions like:

“Why was this alert classified as suspicious?”

A feature-based model can often point to influential terms directly.

Cherry on the cake: Log4Shell showed why text handling matters

A surprising lesson from large-scale text processing came from the Log4Shell vulnerability, tracked as CVE-2021-44228.

The vulnerability affected Apache Log4j and demonstrated that seemingly harmless text input can become dangerous when software interprets it unexpectedly.

The broader lesson for Spark ML engineers is important:

• Treat external text as untrusted data.

• Keep processing dependencies updated.

• Avoid executing or interpreting content from text fields.

• Separate machine learning data pipelines from privileged application logic.

Spark text features are designed to analyze text, not execute it. The security boundary still matters around every system that collects, stores, or processes user-controlled strings.

Production checklist for Spark text features

Before deploying a Spark ML text pipeline:

• Tune numFeatures based on corpus size and accuracy tests.

• Monitor language changes over time.

• Retrain models as vocabulary evolves.

• Validate tokenization with real examples.

• Save complete pipelines instead of individual steps.

• Measure both model quality and processing cost.

Language changes constantly. A model trained on historical messages may degrade when users introduce new products, new terminology, or new attack patterns.

Final thoughts

Text features in Spark provide a practical path from raw language to scalable machine learning.

The core workflow is:

• Use Tokenizer to create terms.

• Use HashingTF to create scalable feature vectors.

• Use IDF to build informative TFIDF representations.

• Use Word2Vec when semantic relationships matter.

• Combine everything in a Spark ML pipeline.

The best next step is hands-on practice: build a small spam classifier using a Tokenizer → HashingTF → IDF → LogisticRegression pipeline, evaluate the results, and then replace or extend the features with Word2Vec embeddings.

Continue to the next lesson or start your own Spark ML text project today—your goal is to turn a real text dataset into a measurable machine learning solution.