-
PySpark ML performance: partitions, data skew, caching, and the small-data trap
PySpark Performance problems are often blamed on cluster size, executor memory, or CPU capacity. Those factors matter, but many slow jobs fail for a simpler reason: the data is distributed badly.
-
From Spark model to production: persistence, MLflow tracking, and batch scoring
A trained Spark ML model is only the beginning of a machine learning lifecycle. In production, the hard problems appear after training:
-
Scaling single-node ML on Spark: pandas API on Spark, applyInPandas, and grouped training
Many machine learning workflows start on one machine:
-
Text features in Spark: Tokenizer, HashingTF, IDF, and Word2Vec on large corpora
Modern machine learning systems often start with a simple challenge: computers do not understand raw text. A support ticket, email, product review, or application log entry is just a sequence of chara
-
Clustering and recommendations at scale: KMeans and ALS collaborative filtering in Spark
Modern applications often need to solve two different machine learning problems:
-
Distributed hyperparameter tuning: CrossValidator, ParamGridBuilder, and TrainValidationSplit
Hyperparameter tuning is often introduced as a modeling task: try different settings, compare metrics, and select the winner. At scale, it becomes a distributed systems problem.
-
Training models with Spark MLlib: LogisticRegression, RandomForest, and GBTClassifier
When your data grows beyond the memory of a single machine, moving it into pandas or NumPy just to train a model becomes expensive. Apache Spark’s spark.ml library keeps feature engineering, model tra
-
Spark ML feature pipelines: VectorAssembler, StringIndexer, OneHotEncoder, and the Pipeline API
Machine learning models rarely consume raw tables directly. Before training, data usually needs cleaning, categorical encoding, feature assembly, and a consistent transformation path for future predic
-
PySpark DataFrames, shuffles, and fast UDFs: pandas_udf + Arrow beat Python UDFs 10x
PySpark is often introduced as “Spark with Python,” but that description hides the most important performance detail: Python is not where Spark does most of its work.
-
Why PySpark for ML: when your data outgrows pandas and a single machine
If you build machine learning systems in Python, pandas is often the first tool you reach for. It is simple, expressive, and excellent for exploration. For many datasets, it is exactly the right choic