-
DuckDB: an analytics warehouse that fits in your laptop’s pocket
If you’ve ever wanted to analyze gigabytes of data without provisioning a database server or signing up for a cloud data warehouse, DuckDB is worth a serious look.
-
ONNX: one model format to run everywhere, from server to browser
Modern AI deployment rarely happens in the same environment where a model was trained.
-
PySpark ML performance: partitions, data skew, caching, and the small-data trap
PySpark Performance problems are often blamed on cluster size, executor memory, or CPU capacity. Those factors matter, but many slow jobs fail for a simpler reason: the data is distributed badly.
-
From Spark model to production: persistence, MLflow tracking, and batch scoring
A trained Spark ML model is only the beginning of a machine learning lifecycle. In production, the hard problems appear after training:
-
Scaling single-node ML on Spark: pandas API on Spark, applyInPandas, and grouped training
Many machine learning workflows start on one machine:
-
Text features in Spark: Tokenizer, HashingTF, IDF, and Word2Vec on large corpora
Modern machine learning systems often start with a simple challenge: computers do not understand raw text. A support ticket, email, product review, or application log entry is just a sequence of chara
-
Clustering and recommendations at scale: KMeans and ALS collaborative filtering in Spark
Modern applications often need to solve two different machine learning problems:
-
Distributed hyperparameter tuning: CrossValidator, ParamGridBuilder, and TrainValidationSplit
Hyperparameter tuning is often introduced as a modeling task: try different settings, compare metrics, and select the winner. At scale, it becomes a distributed systems problem.
-
Training models with Spark MLlib: LogisticRegression, RandomForest, and GBTClassifier
When your data grows beyond the memory of a single machine, moving it into pandas or NumPy just to train a model becomes expensive. Apache Spark’s spark.ml library keeps feature engineering, model tra