Clustering and recommendations at scale: KMeans and ALS collaborative filtering in Spark

LEARN · DISTRIBUTED ML WITH PYSPARK

Modern applications often need to solve two different machine learning problems:

Who are our users? → clustering can discover groups with similar behavior.
What should users see next? → recommendation systems can predict likely preferences.

Apache Spark brings both workflows into a distributed environment through its DataFrame-based spark.ml API. This guide uses current Spark ML patterns available in Apache Spark 4.x releases, including KMeans clustering and ALS collaborative filtering.

We will build two practical pipelines:

A KMeans workflow that assembles customer features and assigns clusters.
An ALS recommender that trains on ratings and handles cold-start cases safely.

The algorithms matter, but production success usually depends on data preparation, evaluation, and operational decisions.

The Spark ML workflow

Spark MLlib provides scalable machine learning algorithms, but the workflow is still familiar:

Load structured data into DataFrames.
Transform raw columns into model-ready features.
Train a distributed model.
Evaluate results.
Generate predictions or recommendations.

A Spark application begins with a Spark session:

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("ClusteringAndRecommendations")
    .getOrCreate()
)

Spark algorithms operate on distributed DataFrames, which means feature engineering is part of the machine learning pipeline rather than a separate local preprocessing step.

KMeans clustering in Spark

What KMeans does

KMeans is an unsupervised learning algorithm that groups similar records into clusters.

The process is:

Choose initial cluster centers.
Assign records to the nearest center.
Recalculate cluster centers.
Repeat until the model converges.

The goal is to create groups where records inside the same cluster are similar.

Common use cases include:

Customer segmentation.
User behavior analysis.
Product grouping.
Exploratory anomaly detection.

For example, an online store might discover groups such as:

Frequent customers with high spending.
Occasional buyers with low activity.
Users who browse often but rarely purchase.

Preparing features with VectorAssembler

Spark ML algorithms expect numerical input in a single vector column. VectorAssembler combines multiple columns into the required format.

Example customer data:

data = [
    (25, 1200.0, 15),
    (31, 5400.0, 42),
    (45, 2300.0, 20),
    (52, 8900.0, 55)
]

columns = [
    "age",
    "annual_spend",
    "purchase_frequency"
]

customers = spark.createDataFrame(data, columns)

customers.show()

Create the feature vector:

from pyspark.ml.feature import VectorAssembler

assembler = VectorAssembler(
    inputCols=[
        "age",
        "annual_spend",
        "purchase_frequency"
    ],
    outputCol="features"
)

feature_df = assembler.transform(customers)

feature_df.select("features").show(truncate=False)

The resulting features column becomes the input for KMeans.

Training a KMeans model

A KMeans model can now assign each customer to a cluster:

from pyspark.ml.clustering import KMeans

kmeans = KMeans(
    k=3,
    seed=42,
    featuresCol="features",
    predictionCol="cluster"
)

kmeans_model = kmeans.fit(feature_df)

clustered = kmeans_model.transform(feature_df)

clustered.select(
    "age",
    "annual_spend",
    "purchase_frequency",
    "cluster"
).show()

The cluster column contains the predicted group for each row.

Choosing the number of clusters

The value of k is one of the biggest decisions in KMeans.

A common measurement is the silhouette score, which estimates how well records fit within their assigned clusters compared with neighboring clusters.

Spark provides a clustering evaluator:

from pyspark.ml.evaluation import ClusteringEvaluator

evaluator = ClusteringEvaluator(
    predictionCol="cluster",
    featuresCol="features",
    metricName="silhouette"
)

score = evaluator.evaluate(clustered)

print(f"Silhouette score: {score}")

A higher score can indicate better-separated clusters, but the best choice depends on the business goal. A marketing team may prefer understandable customer groups, while an operations team may care more about detecting unusual patterns.

Scaling features before clustering

Distance-based algorithms can be heavily influenced by feature ranges.

For example, annual spending values may be thousands while age values are usually much smaller. Without scaling, spending could dominate the distance calculation.

A common approach is standardization:

from pyspark.ml.feature import StandardScaler

scaler = StandardScaler(
    inputCol="features",
    outputCol="scaledFeatures",
    withMean=True,
    withStd=True
)

scaler_model = scaler.fit(feature_df)

scaled_df = scaler_model.transform(feature_df)

In production, teams should monitor:

Cluster sizes over time.
Feature distribution changes.
Users moving between clusters.
Whether clusters remain meaningful.

ALS collaborative filtering for recommendations

How recommendation systems learn preferences

Recommendation engines often learn from behavior instead of manually created rules.

Examples of interaction data:

Movie ratings.
Product purchases.
Music plays.
Article views.

Spark uses Alternating Least Squares (ALS) for matrix-factorization-based recommendation. ALS learns hidden user and item factors from interaction data.

The idea:

Rows represent users.
Columns represent items.
Values represent ratings or interactions.

ALS alternates between optimizing user factors and item factors to find useful patterns.

Preparing ratings data

ALS requires:

A user identifier.
An item identifier.
A rating or interaction value.

Example rating data:

ratings = [
    (1, 101, 5.0),
    (1, 102, 3.0),
    (2, 101, 4.0),
    (2, 103, 5.0),
    (3, 102, 2.0)
]

ratings_df = spark.createDataFrame(
    ratings,
    [
        "userId",
        "movieId",
        "rating"
    ]
)

ratings_df.show()

Real systems usually create these tables from application events, databases, or streaming pipelines.

Training an ALS recommender

Create an ALS model:

from pyspark.ml.recommendation import ALS

als = ALS(
    userCol="userId",
    itemCol="movieId",
    ratingCol="rating",
    rank=10,
    maxIter=10,
    regParam=0.1,
    coldStartStrategy="drop",
    seed=42
)

als_model = als.fit(ratings_df)

The coldStartStrategy="drop" option is important when evaluating models. Unknown users or items can produce invalid predictions, so dropping those rows prevents misleading evaluation results.

Generating recommendations

Spark can recommend items for users:

recommendations = als_model.recommendForAllUsers(5)

recommendations.show(truncate=False)

The output includes recommended item IDs and predicted ratings.

You can also generate item-based recommendations:

item_recommendations = als_model.recommendForAllItems(5)

item_recommendations.show(truncate=False)

Evaluating recommendation quality

For explicit ratings, RMSE is a common offline metric.

from pyspark.ml.evaluation import RegressionEvaluator

train, test = ratings_df.randomSplit(
    [0.8, 0.2],
    seed=42
)

model = als.fit(train)

predictions = model.transform(test)

evaluator = RegressionEvaluator(
    metricName="rmse",
    labelCol="rating",
    predictionCol="prediction"
)

rmse = evaluator.evaluate(predictions)

print(f"RMSE: {rmse}")

A lower RMSE means predictions are closer to known ratings.

However, production recommendation systems usually measure more than prediction accuracy:

Click-through rate.
Conversion rate.
Retention.
Diversity.
Item coverage.
User satisfaction.

A model with slightly better offline accuracy may still create a worse user experience if it recommends repetitive or overly popular items.

The cold-start problem

Collaborative filtering has a fundamental limitation: it needs historical interactions.

New users

A new user has no rating history, so ALS has limited information.

Common solutions:

Recommend popular items.
Ask users for initial preferences.
Combine ALS with content-based approaches.
Use onboarding signals.

New items

A new product, movie, or article has no interaction history.

Possible approaches:

Use item metadata.
Apply business rules.
Blend collaborative filtering with content models.

A common production mistake is evaluating only users and items that already exist in the training data. Offline results may look excellent while real users encounter many new entities.

Cherry on the cake: the Netflix Prize lesson

The Netflix Prize competition, launched in 2006, became a landmark event in recommendation research. Teams competed to improve movie-rating predictions, and leading approaches used advanced matrix-factorization techniques related to modern recommender systems.

The winning team, BellKor’s Pragmatic Chaos, achieved a roughly 10% improvement over Netflix’s original Cinematch baseline.

The surprising lesson was that better prediction accuracy was not enough. Netflix did not simply replace its production recommendation system with the winning algorithm. Real recommendation quality depends on many factors:

Accuracy.
Freshness.
Diversity.
Computing cost.
User experience.

A recommendation engine is not successful because it predicts ratings well in isolation. It succeeds when it helps users discover valuable content.

Combining KMeans and ALS in production

Clustering and recommendations can work together.

A commerce platform could:

Use KMeans to identify behavioral segments.
Train ALS on customer interactions.
Adjust recommendation strategies based on user groups.

A typical architecture looks like:

Spark SQL prepares raw data.
Feature pipelines create ML inputs.
KMeans discovers user segments.
ALS learns personalized recommendations.
A serving layer delivers recommendations.

These models answer different questions:

KMeans: “What type of user is this?”
ALS: “What might this user like next?”

Production checklist

Before deploying Spark ML clustering or recommendation systems, verify:

Training data represents current behavior.
Features are validated and cleaned.
Distance-based models use appropriate scaling.
Cold-start behavior is designed explicitly.
Models are retrained regularly.
Offline metrics match product goals.
Production behavior is monitored.

Spark provides scalable algorithms, but reliable machine learning comes from combining good data engineering with thoughtful product decisions.

Build your own Spark ML project

KMeans and ALS demonstrate two essential machine learning patterns:

Discover hidden structure with clustering.
Predict preferences with collaborative filtering.

Create a Spark project using your own customer or interaction dataset. Build a KMeans segmentation pipeline, add an ALS recommender, test cold-start scenarios, and compare model results against real business goals.

Start your next Spark ML lesson by implementing one complete pipeline, measuring its performance, and sharing what you learned with your team or community.