Polars vs pandas: the DataFrame rewrite that changed the default

For years, the default choice for tabular Python was automatic: import pandas and start manipulating rows. That default has changed.

The useful question is no longer “Which DataFrame library is faster?” It is: Are you exploring an in-memory table, or executing a repeatable analytical query over columnar data?

pandas remains an excellent interactive tool. Polars increasingly becomes the better default for the second workload because it is not merely a faster DataFrame implementation. Its lazy API gives the engine a complete query plan that it can optimize before touching the data.

The comparison is about execution models

pandas is primarily eager. Each operation normally executes and produces an object for the next operation. pandas 3.0 improved its memory semantics substantially: Copy-on-Write is now the default and only mode, removing much of the old view-versus-copy ambiguity. It does not, however, turn a chain of DataFrame calls into a globally optimized query plan. The pandas Copy-on-Write guide explains the new behavior.

Polars can work eagerly too, but its lazy API records expressions until collect() is called. The optimizer can then apply:

• Projection pushdown: scan only columns required downstream.

• Predicate pushdown: move filters toward the data source.

• Slice pushdown: avoid materializing rows outside a requested slice.

• Expression simplification and join planning.

• Common-subplan elimination for reused branches.

• Streaming execution for supported plans.

These are documented optimizer features, not benchmark tricks. Polars also explicitly warns that read_parquet().lazy() is an anti-pattern: the file has already been materialized, so scan-level optimizations are lost. Use scan_parquet() for a lazy pipeline. See the Polars optimizer guide and read_parquet warning.

The same pipeline in both libraries

The benchmark workload reads Parquet, filters recent completed orders, calculates margin, groups by region and channel, aggregates three metrics, and sorts the small result.

pandas receives manual projection and filter pushdown through read_parquet(). That matters: its current API supports columns, filters, and dtype_backend="pyarrow", so omitting them would create an artificially weak baseline. See the pandas 3.0.5 API reference.

from datetime import date
from pathlib import Path

import pandas as pd
import polars as pl

DATA = Path("benchmark-data/orders_5_000_000.parquet")
CUTOFF = date(2026, 1, 1)
COLUMNS = [
    "customer_id",
    "event_date",
    "region",
    "channel",
    "status",
    "revenue",
    "cost",
]


def run_pandas() -> pd.DataFrame:
    frame = pd.read_parquet(
        DATA,
        engine="pyarrow",
        columns=COLUMNS,
        filters=[
            ("event_date", ">=", CUTOFF),
            ("status", "==", "complete"),
            ("revenue", ">=", 250.0),
        ],
        dtype_backend="pyarrow",
    )

    return (
        frame.assign(margin=frame["revenue"] - frame["cost"])
        .groupby(["region", "channel"], as_index=False, sort=False)
        .agg(
            orders=("customer_id", "size"),
            revenue=("revenue", "sum"),
            average_margin=("margin", "mean"),
        )
        .sort_values(
            ["revenue", "region", "channel"],
            ascending=[False, True, True],
            ignore_index=True,
        )
    )


def run_polars() -> pl.DataFrame:
    return (
        pl.scan_parquet(DATA)
        .filter(
            (pl.col("event_date") >= pl.lit(CUTOFF))
            & (pl.col("status") == "complete")
            & (pl.col("revenue") >= 250.0)
        )
        .with_columns(
            (pl.col("revenue") - pl.col("cost")).alias("margin")
        )
        .group_by(["region", "channel"])
        .agg(
            pl.len().cast(pl.Int64).alias("orders"),
            pl.col("revenue").sum(),
            pl.col("margin").mean().alias("average_margin"),
        )
        .sort(
            ["revenue", "region", "channel"],
            descending=[True, False, False],
        )
        .collect(engine="streaming")
    )

The syntax differs, but the decisive distinction is visibility. pandas executes a carefully prepared read followed by eager transformations. Polars sees the full scan-to-aggregation plan before execution and can decide what work is unnecessary.

Use a benchmark that resists easy mistakes

A credible “same pipeline” comparison needs more than two timers around two functions. The accompanying harness deliberately addresses common sources of misleading results:

• It generates deterministic Parquet data with useful columns plus eight unused wide columns.

• The row count is embedded in the filename, such as orders_5_000_000.parquet.

• Stored Parquet metadata is checked before reuse; --regenerate forces a rebuild.

• Each measurement runs in a fresh worker process.

• The filesystem cache is warmed once for each engine.

• Engine order alternates on every repetition to reduce cache and thermal bias.

• Results report median and min–max dispersion, not one lucky run.

• Validation compares every grouped row, keys, ordering, semantic dtypes, revenue, order counts, and average margin with explicit floating-point tolerances.

• Peak RSS uses the operating system’s process high-water mark on POSIX systems; Windows falls back to 5 ms sampling.

Download the complete benchmark harness. It contains generation, version checks, subprocess isolation, validation, timing, and memory reporting without turning this article into a 600-line code listing.

Install the audited stack

As of July 30, 2026, this benchmark pins Python 3.14.6, pandas 3.0.5, Polars 1.43.1, PyArrow 25.0.0, NumPy 2.5.1, and psutil 7.2.2. Python 3.14.6 was released June 10, pandas 3.0.5 on July 22, Polars 1.43.1 on July 27, and Arrow 25.0.0 on July 10. NumPy 2.5.1 supports Python 3.12 through 3.14.

python3.14 -m venv .venv
source .venv/bin/activate
python -m pip install \
  pandas==3.0.5 \
  polars==1.43.1 \
  pyarrow==25.0.0 \
  numpy==2.5.1 \
  psutil==7.2.2

On Windows PowerShell, activate with .venv\Scripts\Activate.ps1.

Run, scale, and inspect

The row-count-specific filename fixes the stale-data bug that occurs when a benchmark silently reuses an earlier orders.parquet file.

# Five million rows, five measured repetitions per engine
python polars_vs_pandas_benchmark.py

# Twenty million rows creates a different validated Parquet file
python polars_vs_pandas_benchmark.py --rows 20_000_000

# Force regeneration of that exact dataset
python polars_vs_pandas_benchmark.py --rows 20_000_000 --regenerate

# Print the optimized Polars plan before running
python polars_vs_pandas_benchmark.py --show-plan

Record the machine, operating system, storage, CPU count, RAM, package versions, dataset size, and complete output. A speedup number without that context is marketing, not an experiment.

Read the result without overgeneralizing

This workload intentionally favors a query engine:

• Columnar Parquet input.

• Selective filters.

• Unused columns that projection can eliminate.

• Vectorized arithmetic.

• Grouped aggregation.

• A small final table.

The gap may shrink when the entire dataset is already hot in cache, almost every row and column is needed, one NumPy kernel dominates, or the result is immediately converted back to pandas. It can also collapse when either pipeline calls Python user-defined functions, because Python becomes the bottleneck.

The gap often grows with wider inputs, more selective filters, multiple joins, and longer transformation chains. That is where avoiding intermediate work matters more than shaving nanoseconds from one arithmetic operation.

Treat memory numbers carefully. Peak RSS includes the Python interpreter, imported native libraries, Arrow buffers, allocator behavior, and compression workspaces. The harness’s POSIX high-water mark avoids missing brief peaks, but it is still process memory—not the exact byte size of a DataFrame. On Windows, the sampling fallback can miss allocations shorter than its 5 ms interval.

Cherry on the cake: lazy execution survived where eager ran out of memory

A May 2026 empirical study ran Polars over TPC-H scale factor 30—about 180 million lineitem rows—on an Azure Standard_DC48s_v3 machine with 48 vCPUs and 384 GiB RAM. Data lived in Azure Blob Storage, and each query configuration was executed five times with the first run excluded as warm-up. The authors tested datasets from roughly 22 GB to 73 GB. Read the paper and methodology.

At the 22 GB configuration, lazy Polars was 2.27× faster than eager execution outside the secure enclave and 2.25× faster inside it. More surprising: eager execution failed with out-of-memory errors at the larger 41 GB and 73 GB configurations, while lazy execution completed them. The paper does not disclose the Polars package version, so this is evidence about that measured system—not a universal promise for Polars 1.43.1.

That result captures the real advantage. Lazy evaluation is not simply deferred syntax. It gives the optimizer opportunities to read fewer columns, apply filters earlier, avoid large intermediates, and stream supported operations. A speedup can therefore come from not doing work, and the same planning can determine whether a pipeline fits in memory at all.

Which library should become your default?

Choose pandas when:

• You are exploring a modest dataset interactively.

• Index alignment is central to the analysis.

• Downstream libraries require pandas objects.

• The workflow contains irregular Python logic.

• Team fluency matters more than pipeline throughput.

Choose lazy Polars when:

• You are building repeatable ETL or analytical jobs.

• Parquet, Arrow, or object storage is the source.

• Filters, joins, and aggregations dominate.

• Data is wide or approaching available memory.

• You want optimization across the complete pipeline.

• Multicore execution and bounded intermediates matter.

Do not rewrite an entire notebook because a benchmark chart looked dramatic. Pick one expensive ingestion-to-aggregation path, implement the same semantics in lazy Polars, validate every output row, and measure repeated wall time plus peak RSS on your own hardware.

Download the harness, replace its synthetic scan with one representative Parquet dataset, and let a reproducible result—not library loyalty—decide your new default.