-
PySpark DataFrames, shuffles, and fast UDFs: pandas_udf + Arrow beat Python UDFs 10x
PySpark is often introduced as “Spark with Python,” but that description hides the most important performance detail: Python is not where Spark does most of its work.