-
ONNX: one model format to run everywhere, from server to browser
Modern AI deployment rarely happens in the same environment where a model was trained.
-
PySpark ML performance: partitions, data skew, caching, and the small-data trap
PySpark Performance problems are often blamed on cluster size, executor memory, or CPU capacity. Those factors matter, but many slow jobs fail for a simpler reason: the data is distributed badly.
-
From Spark model to production: persistence, MLflow tracking, and batch scoring
A trained Spark ML model is only the beginning of a machine learning lifecycle. In production, the hard problems appear after training:
-
Scaling single-node ML on Spark: pandas API on Spark, applyInPandas, and grouped training
Many machine learning workflows start on one machine:
-
Text features in Spark: Tokenizer, HashingTF, IDF, and Word2Vec on large corpora
Modern machine learning systems often start with a simple challenge: computers do not understand raw text. A support ticket, email, product review, or application log entry is just a sequence of chara
-
Clustering and recommendations at scale: KMeans and ALS collaborative filtering in Spark
Modern applications often need to solve two different machine learning problems:
-
Distributed hyperparameter tuning: CrossValidator, ParamGridBuilder, and TrainValidationSplit
Hyperparameter tuning is often introduced as a modeling task: try different settings, compare metrics, and select the winner. At scale, it becomes a distributed systems problem.
-
Training models with Spark MLlib: LogisticRegression, RandomForest, and GBTClassifier
When your data grows beyond the memory of a single machine, moving it into pandas or NumPy just to train a model becomes expensive. Apache Spark’s spark.ml library keeps feature engineering, model tra
-
Spark ML feature pipelines: VectorAssembler, StringIndexer, OneHotEncoder, and the Pipeline API
Machine learning models rarely consume raw tables directly. Before training, data usually needs cleaning, categorical encoding, feature assembly, and a consistent transformation path for future predic
-
PySpark DataFrames, shuffles, and fast UDFs: pandas_udf + Arrow beat Python UDFs 10x
PySpark is often introduced as “Spark with Python,” but that description hides the most important performance detail: Python is not where Spark does most of its work.