-
From Spark model to production: persistence, MLflow tracking, and batch scoring
A trained Spark ML model is only the beginning of a machine learning lifecycle. In production, the hard problems appear after training:
-
Clustering and recommendations at scale: KMeans and ALS collaborative filtering in Spark
Modern applications often need to solve two different machine learning problems:
-
Distributed hyperparameter tuning: CrossValidator, ParamGridBuilder, and TrainValidationSplit
Hyperparameter tuning is often introduced as a modeling task: try different settings, compare metrics, and select the winner. At scale, it becomes a distributed systems problem.
-
Training models with Spark MLlib: LogisticRegression, RandomForest, and GBTClassifier
When your data grows beyond the memory of a single machine, moving it into pandas or NumPy just to train a model becomes expensive. Apache Spark’s spark.ml library keeps feature engineering, model tra