-
Living with pyspark.pandas: interop, performance traps, and a pandas-to-Spark migration checklist
Lesson 11 mapped what the pandas API on Spark implements; lesson 12 proved on 103 million rows that the port can be a handful of changed lines. This closing lesson is about operating it: moving data b
-
One dataset, two engines: scaling California housing until pandas gives up, then rerunning it on pyspark.pandas
Lesson 11 mapped what the pandas API on Spark implements. This lesson measures. We take a dataset every DS course uses — scikit-learn’s California housing, 20,640 rows — and multiply it with jitter un
-
pandas API on Spark in depth: what pyspark.pandas covers, what it doesn’t, and how it executes
You already know the promise from earlier lessons in this track: when a dataset outgrows one machine, Spark distributes the work. The catch has always been the rewrite — a pandas notebook does not bec
-
Scaling single-node ML on Spark: pandas API on Spark, applyInPandas, and grouped training
Many machine learning workflows start on one machine: