pandas API on Spark in depth: what pyspark.pandas covers, what it doesn’t, and how it executes

You already know the promise from earlier lessons in this track: when a dataset outgrows one machine, Spark distributes the work. The catch has always been the rewrite — a pandas notebook does not become a pyspark.sql.DataFrame program by search-and-replace. The pandas API on Spark (import pyspark.pandas as ps) is Spark’s answer: a pandas-shaped layer over Spark DataFrames that lets most pandas code run distributed with minimal edits. Lesson 8 introduced it in passing; this lesson is the deep dive. What is actually implemented, what is deliberately missing, what behaves differently — and the execution model you must understand before trusting it with production work.

Where it came from, and where it is now

The project started life outside Spark as Koalas, a Databricks open-source library (2019) that reimplemented the pandas API over Spark DataFrames. In Spark 3.2 (2021) Koalas was merged into Spark itself as pyspark.pandas, and the standalone package was frozen. Since then it has shipped with every Spark release — no extra install, no version skew against your cluster.

The practical state today, on Spark 4.2 (released July 2026):

• Spark 4.x requires pandas ≥ 2.2 on the driver and workers (raised from 2.0 in Spark 4.1) and aligns its semantics with pandas 2.x — the pandas-1.x-era behaviors (iteritems, append, default numeric_only quirks) are gone, matching what modern pandas itself did.
• The API surface tracks pandas closely enough that the official docs maintain a per-method support matrix; the majority of the day-to-day DataFrame/Series/GroupBy API is implemented.
• It interoperates with the rest of Spark: a ps.DataFrame converts to and from a native Spark DataFrame without copying data (.to_spark() / .pandas_api()), so you can drop to Spark SQL for one step and come back.

One sentence of positioning against lesson 8: applyInPandas distributes your pandas functions over groups; pyspark.pandas replaces the pandas API itself with a distributed implementation. They compose — you will often use both in one job.

The execution model: eager-looking, lazy underneath

Every ps.DataFrame wraps a Spark logical plan plus index metadata (internally, an InternalFrame). When you write:

import pyspark.pandas as ps

df = ps.read_parquet("s3://bucket/housing/")
df["rooms_per_person"] = df["AveRooms"] / df["AveOccup"]
grid = df.groupby(["lat_cell", "lon_cell"])["MedHouseVal"].mean()

no data has been processed yet. Each line extends a Spark plan. Execution happens when something needs actual values: .head(), len(df), .to_pandas(), .plot(), writing output, or printing the frame in a notebook. This is the single biggest mental-model shift from pandas, and it cuts both ways:

• For you: Spark’s optimizer sees your whole chain of operations and can fuse, reorder, and prune columns before touching data. Ten pandas-style assignments cost one pass, not ten.
• Against you: every materialization re-runs the plan unless you cache. A notebook that displays df after each step re-executes the growing plan each time. Errors also surface late — a bad cast in step 2 explodes at the .head() in step 9, with a Spark stack trace.

Two idioms follow directly. First, checkpoint long chains:

df = df.spark.cache()        # persist the current plan's result
# ...or cut the lineage entirely after a very long chain:
df = df.spark.local_checkpoint()

Second, when a notebook feels “slow at random lines”, it is not random — those are the action lines. Profile by looking for materialization points, not by timing assignments.

The index problem: the one design decision to understand

pandas guarantees every frame an index — an ordered row-label structure. Distributed data has no inherent order, so pyspark.pandas must manufacture an index when you don’t supply one. The compute.default_index_type option picks the strategy, and choosing wrong is the #1 silent performance killer:

Index type	How it works	Guarantees	Cost
sequence	builds 0,1,2,… with a window over one partition	exact pandas semantics, deterministic	gathers all rows through a single partition — cluster reduced to one lane
distributed-sequence (default)	computes global 0,1,2,… by counting partition sizes first	continuous increasing ids	an extra job/shuffle before many operations; ids can change between operations that repartition
distributed	monotonically increasing but non-continuous ids, purely local per partition	no shuffle, no extra pass — fast	ids have gaps, differ run to run; positional semantics are gone

ps.set_option("compute.default_index_type", "distributed")  # fast, weakest semantics

The right habit is better than any default: carry a real index. Pass index_col= on read_parquet/read_csv/to_spark, keep a meaningful key column, and you avoid manufactured indexes entirely:

df = ps.read_parquet("s3://bucket/housing/", index_col="listing_id")

If you profile a pyspark.pandas job and see an unexplained first stage touching every row, it is almost always the default index being built. Lesson 12 measures this on a 103M-row workflow — with a finding worth spoiling: the tax is only charged when something actually materializes the index. A pipeline of pure column operations, groupbys, and merges never builds it at all, and the three index types time identically; the moment an operation needs row identity, the differences appear, and sequence in particular collapses the job onto one partition.

Cross-frame operations: the ops_on_diff_frames guard (now off by default)

In pandas, df1["a"] + df2["b"] silently aligns on the index. Distributed, that alignment is a join — potentially a massive shuffle you didn’t ask for. For years pyspark.pandas refused such operations unless you opted in via compute.ops_on_diff_frames; tutorials still describe that behavior. On Spark 4.2 the default has flipped: the option ships as True, and cross-frame arithmetic silently executes as an index join.

ps.get_option("compute.ops_on_diff_frames")   # True on Spark 4.2 — verified
combined = df1["a"] + df2["b"]                # runs; behind it: a join on the index

Convenient — and exactly the kind of convenience that hides five shuffles in one notebook cell. The defensive habit is to turn the guard back on at session start:

ps.set_option("compute.ops_on_diff_frames", False)
df["n"] = other_frame["count"]
# ValueError: Cannot combine the series or dataframe because it comes from a
# different dataframe. In order to allow this operation, enable 'compute.ops_on_diff_frames'

and then express every intended alignment as an explicit merge/join on a real key, which states the join you meant. Same-frame operations (df["a"] + df["b"]) never involve this and never shuffle.

Coverage map: what works today

The honest headline: most day-to-day pandas code runs unchanged. Concretely, on Spark 4.2:

Area	Status in pyspark.pandas
IO	read_parquet, read_csv, read_json, read_orc, read_delta, read_sql, read_excel, read_html, read_clipboard; matching writers
Column arithmetic & comparisons	full, including NumPy ufuncs (np.log1p(df["Population"]) works)
Selection	[], .loc, .iloc (with caveats below), boolean masks, query, filter, where, mask, isin
groupby	agg/dict-agg, mean/sum/count/min/max/std/var/median, nunique, size, first/last, transform, apply, filter, cumcount, rank, head, shift, fillna
Joins	merge, join — executed as Spark joins with all its strategies (broadcast, sort-merge)
Reshaping	concat, pivot_table (requires an explicit columns=), melt, stack/unstack, get_dummies, explode, transpose (guarded)
Missing data	isna, fillna, dropna, replace, interpolate (limited methods)
Datetime / string accessors	.dt.* and .str.* largely complete (implemented over Spark SQL functions)
Window	rolling and expanding — sum/mean/min/max/count/std/var/skew/kurt/quantile; ewm with mean
Stats	describe, corr, cov, quantile (approximate by default), value_counts, nlargest/nsmallest, rank
Plotting	.plot backed by plotly by default — histograms, box, scatter, line; heavy plots downsample/aggregate on the cluster first
Interop	to_pandas, to_numpy (both collect!), to_spark, pandas_api, apply_batch/transform_batch for chunked custom pandas code

Two structural features are genuinely there and often assumed missing: MultiIndex (including groupby producing one, stack, and index levels) and categorical dtype (astype("category"), .cat accessor). Both have holes at the edges, but their cores work.

Coverage map: what’s missing, guarded, or different

This is the part the marketing page won’t tell you. Four buckets.

1. Not implemented (you get PandasNotImplementedError).

• ps.cut and ps.qcut exist as names but are stubs — calling either raises PandasNotImplementedError: The method pd.cut() is not implemented yet (verified on 4.2). Bucket with Bucketizer/SQL width_bucket via .to_spark(), or apply_batch.
• Series.interpolate beyond linear on numerics, parts of the .cat/.dt edges, DataFrame.lookup-style APIs already deprecated by pandas itself.
• .style, memory_usage(), exotic reshape corners, parts of resample (a subset of frequencies/aggregations works — daily/rule-based resampling does).
• Cell-level writes: df.at[i, c] = x / df.iat assignment raise TypeError — there is no distributed single-cell mutation. Column-level assignment and insert work fine.

2. Implemented but guarded, because it can kill your cluster.

• transpose() and to_pandas()-adjacent conveniences check compute.max_rows (default 1000) and refuse above it unless you raise the option — a transpose of a billion-row frame is a collect in disguise.
• sort_values works but is a full shuffle; sort_index on a manufactured index may first build that index. Sorting “just to look at the data” costs a job.
• .iloc supports slicing (df.iloc[:1000, [0, 2]]) and, on 4.2, arbitrary row lists — but df.iloc[[3, 1]] comes back in index order (1, 3), not your requested order (verified). Positional-order semantics need a global order that distributed data doesn’t have; anything downstream that relies on the list’s order breaks silently.

3. Same API, different behavior — the dangerous bucket.

• Row order is not preserved. After any shuffle-inducing operation, head(5) returns a five rows, not the first five of your mental model. Anything that implicitly relied on file order in pandas needs an explicit sort_values.
• apply/transform with non-trivial functions execute your Python per batch via Arrow, and the function is evaluated once on a sample to infer the return schema — a function with side effects or sample-sensitive types will surprise you. Annotate return types (def f(s) -> ps.Series[float]:) to skip inference.
• NaN vs null: Spark has one null; pandas distinguishes NaN/None/NaT and (in 2.x) pd.NA. Round-tripping through to_pandas() can change dtypes (int column with nulls arrives as float or nullable Int depending on Arrow settings).
• inplace=True is accepted on many methods but is a client-side convenience: it rebinds the wrapped plan. It never saves memory the way pandas users hope, and mutating a frame that another variable also references does not propagate the way pandas views sometimes do.
• corr/quantile default to approximate distributed algorithms; exact quantiles on billions of rows are a deliberate non-goal. describe() inherits this.
• Duplicate index labels are legal but many alignment operations degrade or refuse; pandas tolerates them more broadly.

4. Performance semantics, not correctness.

• Every column assignment extends the plan; hundreds of sequential assignments (feature-factory loops) build plans the optimizer chokes on. Batch them or checkpoint periodically.
• for row in df.iterrows() exists and is a catastrophe at scale — it collects. The presence of an API does not make it distributed-appropriate.

The options that matter

All tunable via ps.set_option / ps.option_context:

ps.set_option("compute.default_index_type", "distributed")   # see index table above
ps.set_option("compute.ops_on_diff_frames", False)           # restore the cross-frame guard (4.2 ships True)
ps.set_option("compute.max_rows", 1000)                      # guard for transpose & friends (default 1000)
ps.set_option("plotting.max_rows", 1000)                     # plot downsampling threshold (default 1000)
ps.set_option("compute.shortcut_limit", 1000)                # small-frame fast path threshold (default 1000)

compute.shortcut_limit is worth knowing: below it, some operations collect and compute in plain pandas because that’s faster than a Spark job — an admission of the small-data trap from lesson 10, built into the library.

What to take away

pyspark.pandas is not a compatibility shim; it is a second implementation of the pandas API with distributed semantics, and the differences are principled: everything that would require a global order or a full collect is either manufactured (indexes), guarded (transpose), silently reordered (iloc lists), executed as a hidden join (cross-frame ops, now on by default), or intentionally approximate (quantiles). Learn the four buckets above and 90% of a working pandas notebook ports by changing one import.

The next lesson stops talking and measures: the same EDA-and-features workflow on 20 thousand rows, then multiplied out to 100 million — plain pandas until it dies, then pyspark.pandas on identical code, with wall-clock and memory numbers for every rung.