-
Living with pyspark.pandas: interop, performance traps, and a pandas-to-Spark migration checklist
Lesson 11 mapped what the pandas API on Spark implements; lesson 12 proved on 103 million rows that the port can be a handful of changed lines. This closing lesson is about operating it: moving data b
-
One dataset, two engines: scaling California housing until pandas gives up, then rerunning it on pyspark.pandas
Lesson 11 mapped what the pandas API on Spark implements. This lesson measures. We take a dataset every DS course uses — scikit-learn’s California housing, 20,640 rows — and multiply it with jitter un
-
pandas API on Spark in depth: what pyspark.pandas covers, what it doesn’t, and how it executes
You already know the promise from earlier lessons in this track: when a dataset outgrows one machine, Spark distributes the work. The catch has always been the rewrite — a pandas notebook does not bec
-
PySpark ML performance: partitions, data skew, caching, and the small-data trap
PySpark Performance problems are often blamed on cluster size, executor memory, or CPU capacity. Those factors matter, but many slow jobs fail for a simpler reason: the data is distributed badly.
-
Scaling single-node ML on Spark: pandas API on Spark, applyInPandas, and grouped training
Many machine learning workflows start on one machine:
-
PySpark DataFrames, shuffles, and fast UDFs: pandas_udf + Arrow beat Python UDFs 10x
PySpark is often introduced as “Spark with Python,” but that description hides the most important performance detail: Python is not where Spark does most of its work.
-
Why PySpark for ML: when your data outgrows pandas and a single machine
If you build machine learning systems in Python, pandas is often the first tool you reach for. It is simple, expressive, and excellent for exploration. For many datasets, it is exactly the right choic