Parquet and Arrow: why columnar formats make your queries 100x faster

Large analytical datasets often start life as CSV files. They are easy to create, easy to open, and supported almost everywhere.

The problem appears when the data grows.

A CSV file stores information row by row:

Alice,29,London,Engineer
Bob,41,Berlin,Designer
Carol,35,Paris,Manager

If a query only needs the age column, a CSV reader still has to:

• Read each row.

• Parse separators and text values.

• Decode columns that will never be used.

• Discard unnecessary data.

For small files, this does not matter. For gigabytes or terabytes of analytical data, the wasted work becomes expensive.

Columnar storage changes the layout completely. Instead of storing complete records together, it stores values from the same column together:

name: Alice, Bob, Carol
age: 29, 41, 35
city: London, Berlin, Paris
job: Engineer, Designer, Manager

This simple design choice powers much of modern analytics. Apache Parquet and Apache Arrow are two technologies built around this idea, but they solve different problems.

In this guide, you will learn:

• How Parquet stores analytical data efficiently.

• Why Arrow improves in-memory processing.

• How column pruning and predicate pushdown accelerate queries.

• How compression works better with column-oriented data.

• How to benchmark CSV versus Parquet yourself.

• Why some workloads can become dramatically faster.

Parquet and Arrow solve different problems

Although Parquet and Arrow are often mentioned together, they operate at different layers.

Apache Parquet: storage optimized for analytics

Parquet is a columnar file format designed for storing data.

Its strengths include:

• Column-oriented layout.

• Efficient compression.

• Embedded schemas.

• Support for analytical query engines.

• Cross-language compatibility.

Think of Parquet as an optimized alternative to CSV for datasets that will be queried repeatedly.

Apache Arrow: memory optimized for data exchange

Arrow is an in-memory columnar format.

It focuses on moving and processing data efficiently between systems such as:

• Python.

• Rust.

• Java.

• C++.

• Data processing libraries.

• Query engines.

Arrow avoids many expensive conversions by representing data in contiguous memory buffers rather than collections of individual objects.

A useful mental model:

• Parquet = optimized for storing data on disk.

• Arrow = optimized for representing data in memory.

Arrow is a common interchange format in modern data systems, but it is not a requirement for every execution engine. Some engines use Arrow internally, while others use their own columnar memory formats or only convert data into Arrow structures at specific boundaries.

Why columnar storage is faster

Imagine a user table with 150 columns.

The query is:

SELECT age
FROM users
WHERE age > 30;

A traditional row-oriented reader may need to process every field in every record.

A Parquet reader can often read only the columns needed for the query.

This technique is called column pruning.

The benefits are significant:

• Less disk I/O.

• Less CPU spent parsing data.

• Lower memory usage.

• Faster query execution.

For wide analytical datasets, avoiding unnecessary columns is often the biggest performance improvement.

Row groups: the building blocks behind fast scans

Parquet files are not simply one giant block of columns. They are organized into structures called row groups.

A simplified layout looks like this:

Parquet file

Row group 1
  age column
  country column
  salary column

Row group 2
  age column
  country column
  salary column

Row group 3
  age column
  country column
  salary column

Each row group can contain metadata describing its contents.

For example:

age column statistics

Row group 1:
minimum = 18
maximum = 25

Row group 2:
minimum = 40
maximum = 65

If a query asks:

SELECT *
FROM users
WHERE age > 50;

The engine can skip row group 1 because it knows that group cannot contain matching values.

This optimization is called predicate pushdown.

Instead of:

1. Read everything.

2. Filter later.

The engine can do:

1. Check metadata.

2. Skip irrelevant data.

3. Read only useful sections.

Why Parquet compresses so well

Compression works best when similar values are stored together.

Rows mix unrelated information:

Alice,29,London
Bob,41,Berlin
Carol,35,Paris

Columns group similar information:

29
41
35

or:

London
London
London
London

Parquet can apply specialized encodings before compression, including:

• Dictionary encoding for repeated values.

• Run-length encoding for repeated sequences.

• Bit packing for compact numeric storage.

• Delta encoding for values that change predictably.

After encoding, Parquet commonly uses compression codecs such as:

• Zstandard.

• Snappy.

• Gzip.

• Brotli.

The combination of column layout and encoding is why analytical datasets often become much smaller than their CSV equivalents.

Arrow keeps analytical data CPU-friendly

Many analytical operations are faster when values are stored in contiguous memory buffers.

Traditional Python objects can carry significant overhead because each value may be a separate object managed by the runtime.

Arrow arrays instead store data in structures designed for efficient processing.

Advantages include:

• Better CPU cache usage.

• Efficient vectorized operations.

• Lower memory overhead.

• Faster interchange between compatible tools.

When Arrow is preferable to a DataFrame workflow depends on the task. For interactive analysis, Pandas is often convenient. For large-scale pipelines, streaming, cross-language processing, or moving data between engines, Arrow can be a better fit.

A runnable CSV vs. Parquet benchmark

The following benchmark creates a dataset, writes both formats, compares file sizes, and measures scan performance.

Install current packages:

pip install -U "pandas>=2.2" "pyarrow>=21.0.0" "numpy>=2.0"

Create the dataset:

import numpy as np
import pandas as pd

rows = 2_000_000

rng = np.random.default_rng(42)

df = pd.DataFrame(
    {
        "id": np.arange(rows),
        "country": rng.choice(
            ["US", "UK", "DE", "FR", "JP", "CA"],
            rows,
        ),
        "age": rng.integers(18, 80, rows),
        "salary": rng.normal(70000, 15000, rows).round(2),
        "score": rng.random(rows),
    }
)

df.to_csv("data.csv", index=False)

df.to_parquet(
    "data.parquet",
    compression="zstd",
)

Compare storage size:

from pathlib import Path

csv_size = Path("data.csv").stat().st_size / 1024 / 1024
parquet_size = Path("data.parquet").stat().st_size / 1024 / 1024

print(f"CSV: {csv_size:.1f} MB")
print(f"Parquet: {parquet_size:.1f} MB")
print(f"Compression ratio: {csv_size / parquet_size:.2f}x")

A typical result might show several times less storage for Parquet. Your result will depend on:

• Data distribution.

• Compression settings.

• Column types.

• Dataset size.

Now compare reading the files:

import time
import pandas as pd

start = time.perf_counter()

csv_df = pd.read_csv("data.csv")
csv_df["age"].mean()

csv_time = time.perf_counter() - start

start = time.perf_counter()

parquet_df = pd.read_parquet("data.parquet")
parquet_df["age"].mean()

parquet_time = time.perf_counter() - start

print(f"CSV scan: {csv_time:.3f}s")
print(f"Parquet scan: {parquet_time:.3f}s")

Finally, test column pruning:

import time
import pandas as pd
import pyarrow.parquet as pq

start = time.perf_counter()

ages = pd.read_csv(
    "data.csv",
    usecols=["age"],
)

ages["age"].mean()

csv_column_time = time.perf_counter() - start

start = time.perf_counter()

table = pq.read_table(
    "data.parquet",
    columns=["age"],
)

table.column("age").to_numpy().mean()

parquet_column_time = time.perf_counter() - start

print(f"CSV single column: {csv_column_time:.3f}s")
print(f"Parquet single column: {parquet_column_time:.3f}s")

When can Parquet become 100× faster?

The phrase “100× faster” can happen, but it is not a universal Parquet benchmark result.

Large speedups usually happen when several optimizations combine:

• Only a small number of columns are needed.

• Predicate pushdown removes most row groups.

• Compression reduces storage reads.

• The engine executes operations in parallel.

• Vectorized processing avoids row-by-row work.

Consider a dataset with:

• 150 columns.

• Billions of rows.

• A query needing two columns.

• A filter that eliminates most data.

The engine may read a tiny fraction of the original dataset.

That can create order-of-magnitude improvements.

However, scanning every column of every row is a different workload. Parquet still helps through compression and efficient storage, but the speed difference will usually be smaller.

Cherry on the cake: a taxi dataset that shrank dramatically

A famous real-world example comes from the public New York City Taxi and Limousine Commission trip records.

The raw taxi trip data is published as CSV files. Data engineers frequently convert these files into Parquet for analytics workflows.

One commonly reproduced conversion of a monthly yellow taxi dataset reduced storage from roughly 3 GB of CSV data to around 500 MB of Parquet data using columnar encoding and compression.

The surprising part is that the compression codec is only part of the story. The column layout itself creates the conditions for better compression:

• Passenger counts repeat.

• Payment types repeat.

• Location identifiers repeat.

• Numeric columns often have predictable patterns.

Parquet does not just compress the file better; it organizes the information so compression algorithms have much more useful patterns to work with.

When should you use Parquet?

Parquet is a strong choice for:

• Data lakes.

• Business intelligence systems.

• Machine learning datasets.

• ETL pipelines.

• Historical analytics.

• Batch processing.

CSV remains useful when:

• Humans need to inspect files manually.

• A simple interchange format is required.

• Maximum compatibility matters more than performance.

For modern ColumnarStorage workflows, Parquet is often the default format because it balances speed, size, and interoperability.

Best practices

A few habits provide most of the benefits:

• Store analytical datasets as Parquet instead of repeatedly processing CSV.

• Read only the columns required by your query.

• Use appropriate compression, such as Zstandard, when supported by your tools.

• Keep schemas consistent across files.

• Partition very large datasets carefully.

• Avoid unnecessary conversions between formats.

• Use Arrow-compatible tools when moving large amounts of data between systems.

Key takeaways

Remember these fundamentals:

• CSV stores complete rows; Parquet stores columns.

• Parquet optimizes storage; Arrow optimizes memory representation and data exchange.

• Column pruning avoids unnecessary reads.

• Predicate pushdown skips irrelevant data.

• Column-oriented layouts improve compression.

• The biggest speedups come from reducing the amount of data processed.

Try the benchmark on your own dataset, compare your CSV and Parquet results, and measure how much storage and query time you can save. Follow the next lessons in this course to keep building practical data engineering skills.