-
Spark ML feature pipelines: VectorAssembler, StringIndexer, OneHotEncoder, and the Pipeline API
Machine learning models rarely consume raw tables directly. Before training, data usually needs cleaning, categorical encoding, feature assembly, and a consistent transformation path for future predic
-
PySpark DataFrames, shuffles, and fast UDFs: pandas_udf + Arrow beat Python UDFs 10x
PySpark is often introduced as “Spark with Python,” but that description hides the most important performance detail: Python is not where Spark does most of its work.
-
Why PySpark for ML: when your data outgrows pandas and a single machine
If you build machine learning systems in Python, pandas is often the first tool you reach for. It is simple, expressive, and excellent for exploration. For many datasets, it is exactly the right choic