-
Living with pyspark.pandas: interop, performance traps, and a pandas-to-Spark migration checklist
Lesson 11 mapped what the pandas API on Spark implements; lesson 12 proved on 103 million rows that the port can be a handful of changed lines. This closing lesson is about operating it: moving data b
-
One dataset, two engines: scaling California housing until pandas gives up, then rerunning it on pyspark.pandas
Lesson 11 mapped what the pandas API on Spark implements. This lesson measures. We take a dataset every DS course uses — scikit-learn’s California housing, 20,640 rows — and multiply it with jitter un
-
pandas API on Spark in depth: what pyspark.pandas covers, what it doesn’t, and how it executes
You already know the promise from earlier lessons in this track: when a dataset outgrows one machine, Spark distributes the work. The catch has always been the rewrite — a pandas notebook does not bec
-
PySpark ML performance: partitions, data skew, caching, and the small-data trap
PySpark Performance problems are often blamed on cluster size, executor memory, or CPU capacity. Those factors matter, but many slow jobs fail for a simpler reason: the data is distributed badly.
-
From Spark model to production: persistence, MLflow tracking, and batch scoring
A trained Spark ML model is only the beginning of a machine learning lifecycle. In production, the hard problems appear after training:
-
Scaling single-node ML on Spark: pandas API on Spark, applyInPandas, and grouped training
Many machine learning workflows start on one machine:
-
Text features in Spark: Tokenizer, HashingTF, IDF, and Word2Vec on large corpora
Modern machine learning systems often start with a simple challenge: computers do not understand raw text. A support ticket, email, product review, or application log entry is just a sequence of chara
-
Clustering and recommendations at scale: KMeans and ALS collaborative filtering in Spark
Modern applications often need to solve two different machine learning problems:
-
Distributed hyperparameter tuning: CrossValidator, ParamGridBuilder, and TrainValidationSplit
Hyperparameter tuning is often introduced as a modeling task: try different settings, compare metrics, and select the winner. At scale, it becomes a distributed systems problem.
-
Training models with Spark MLlib: LogisticRegression, RandomForest, and GBTClassifier
When your data grows beyond the memory of a single machine, moving it into pandas or NumPy just to train a model becomes expensive. Apache Spark’s spark.ml library keeps feature engineering, model tra