Chunking strategies that actually matter for retrieval quality

LEARN · RAG & AGENTIC DEVELOPMENT

Why chunking deserves more attention

If you’ve experimented with embedding models, vector databases, rerankers, or prompts, but your Retrieval-Augmented Generation (RAG) system still misses obvious answers, the bottleneck may be much simpler: chunking.

Chunking determines the units your retriever can return. If the chunks are poorly constructed, retrieval quality suffers before the language model ever sees the context.

This guide explains the chunking strategies that have the biggest practical impact, when to use each one, how to evaluate them, and includes a runnable comparison between fixed-size and paragraph-aware chunking. The emphasis is on measurable retrieval quality rather than intuition.

Before documents are embedded, they are split into smaller retrieval units called chunks.

Instead of generating one embedding for an entire document, you generate many embeddings—one per chunk. During retrieval, the vector index searches those chunk embeddings.

The goal is simple:

Each chunk should represent one coherent topic.
Each chunk should contain enough context to answer a question independently.
Each chunk should avoid mixing unrelated ideas.

Good chunking improves:

Recall
Ranking quality
Context precision
Downstream answer quality

Poor chunking can make an excellent embedding model appear ineffective.

These two approaches are often confused.

Character-based chunking

Character-based chunking splits after a fixed number of characters.

Advantages:

Extremely simple
Very fast
No tokenizer dependency

Disadvantages:

Doesn’t correspond to embedding model tokenization
May exceed token limits unexpectedly
Can split sentences awkwardly

Token-based chunking

Token-based chunking measures chunk size using the tokenizer associated with the embedding model.

Advantages:

Predictable embedding sizes
Better alignment with model context limits
More consistent retrieval behavior across documents

For production RAG systems, token-based chunking is generally the preferred baseline because embedding models consume tokens rather than characters.

Throughout this article, the educational chunking implementations use character counts to keep the examples dependency-light and easy to understand. In production, replace those character limits with token counts using the tokenizer for your chosen embedding model.

Imagine a documentation page with sections covering:

Installation
Authentication
Rate limits
Error handling
Pricing

A user asks:

How do I authenticate?

If all five sections exist inside one chunk, the embedding represents several unrelated concepts simultaneously.

Now consider the opposite extreme:

“To authenticate”
“send your API key”
“inside the Authorization header”

Those chunks are so small that none contains enough information on its own.

The sweet spot lies between these extremes.

Well-designed chunks are:

Self-contained
Focused on one topic
Large enough to preserve meaning
Small enough to avoid unrelated information

Fixed-size chunking splits text after a predetermined number of characters or tokens.

It remains one of the strongest baselines because it is predictable, easy to implement, and scales well.

from pathlib import Path

text = Path("document.txt").read_text(encoding="utf-8")

CHUNK_SIZE = 800

chunks = [
    text[i:i + CHUNK_SIZE]
    for i in range(0, len(text), CHUNK_SIZE)
]

print(f"Created {len(chunks)} chunks")
print(chunks[0][:250])

Best for

Log files
Large text corpora
OCR output
Chat exports
Uniformly formatted documents

Drawbacks

May split sentences
Ignores document structure
Can separate explanations from examples
May cut through code blocks

Despite those limitations, fixed-size chunking is an excellent benchmark because every more sophisticated strategy should outperform it on retrieval metrics.

A simple improvement is respecting paragraph boundaries.

Instead of cutting arbitrarily, paragraphs are grouped until a target size is reached.

This preserves much more of the author’s intended structure.

from pathlib import Path

text = Path("document.txt").read_text(encoding="utf-8")

paragraphs = [
    p.strip()
    for p in text.split("\n\n")
    if p.strip()
]

MAX_LENGTH = 1200

chunks = []
current = []

for paragraph in paragraphs:
    if len(paragraph) > MAX_LENGTH:
        if current:
            chunks.append("\n\n".join(current))
            current = []

        for i in range(0, len(paragraph), MAX_LENGTH):
            chunks.append(paragraph[i:i + MAX_LENGTH])

        continue

    candidate = "\n\n".join(current + [paragraph])

    if len(candidate) <= MAX_LENGTH:
        current.append(paragraph)
    else:
        if current:
            chunks.append("\n\n".join(current))
        current = [paragraph]

if current:
    chunks.append("\n\n".join(current))

print(f"Created {len(chunks)} paragraph-aware chunks")

Although this example measures size in characters for simplicity, the same algorithm can use token counts in production.

Paragraph-aware chunking often improves retrieval for documentation, blogs, and reports because paragraphs usually correspond to coherent ideas.

Many production RAG pipelines go one step further by preserving document structure instead of relying purely on size.

Examples include:

Content	Preferred boundary
Markdown	Headings
HTML	Sections
API documentation	Endpoints
Source code	Functions and classes
Books	Chapters and sections
Chat transcripts	Speaker turns
JSON	Complete objects
SQL	Individual statements

The guiding principle is simple:

Keep related information together because users tend to ask questions about logical units, not arbitrary slices of text.

One of the easiest ways to reduce retrieval quality is breaking code in the middle of a function.

Avoid creating chunks like this:

Chunk A

def authenticate():
    token = create_token()

Chunk B

    return token

Instead, keep the entire function intact.

The same recommendation applies to:

JSON
YAML
SQL
XML
Configuration files
Markdown code fences

Broken syntax weakens embeddings because meaningful relationships disappear when code is fragmented.

Many chunkers duplicate part of the previous chunk into the next one.

For example:

Chunk 1 contains lines 1–100
Chunk 2 contains lines 91–190

The duplicated region helps when relevant information sits near a boundary.

A practical starting point is:

500–1,000 tokens per chunk
Around 10–20% overlap

These are starting points rather than universal rules.

A surprising result from many RAG evaluations is that increasing overlap indefinitely rarely improves retrieval.

Large overlaps create many nearly identical embeddings.

Instead of retrieving complementary evidence, the retriever may return multiple copies of essentially the same passage.

The result can include:

Larger vector indexes
Higher embedding costs
Slower ingestion
Less diverse retrieved context

If retrieval quality is disappointing, improving chunk boundaries is often more effective than continuously increasing overlap.

Rather than comparing chunk counts, compare retrieval quality.

The example below includes a tiny self-contained document so it runs without requiring external files.

import numpy as np
from sentence_transformers import SentenceTransformer

DOCUMENT = """
# Installation

Install the package using pip.

# Authentication

Authenticate by sending your API key in the Authorization header.

# Rate limits

The API allows 100 requests per minute.

# Error handling

Errors are returned as JSON with an HTTP status code.
"""

queries = [
    ("How do I authenticate?", "Authorization header"),
    ("What are the rate limits?", "100 requests per minute"),
    ("How are errors returned?", "JSON")
]


def fixed_chunks(text, size=120):
    return [
        text[i:i + size]
        for i in range(0, len(text), size)
    ]


def paragraph_chunks(text, max_length=180):
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]

    chunks = []
    current = []

    for paragraph in paragraphs:
        candidate = "\n\n".join(current + [paragraph])

        if len(candidate) <= max_length:
            current.append(paragraph)
        else:
            chunks.append("\n\n".join(current))
            current = [paragraph]

    if current:
        chunks.append("\n\n".join(current))

    return chunks


model = SentenceTransformer("BAAI/bge-small-en-v1.5")


def recall_at_k(chunks, k=3):
    embeddings = model.encode(chunks, normalize_embeddings=True)
    hits = 0

    for query, expected in queries:
        q = model.encode(query, normalize_embeddings=True)
        scores = embeddings @ q
        top = np.argsort(scores)[::-1][:k]

        if any(expected.lower() in chunks[i].lower() for i in top):
            hits += 1

    return hits / len(queries)


fixed = fixed_chunks(DOCUMENT)
paragraph = paragraph_chunks(DOCUMENT)

print("Fixed Recall@3:", recall_at_k(fixed))
print("Paragraph Recall@3:", recall_at_k(paragraph))

This benchmark is intentionally small, but it demonstrates the correct evaluation mindset:

Measure retrieval quality instead of assuming one strategy is better.

Chunking should be evaluated like any other retrieval component.

A practical workflow looks like this:

Build a representative set of user questions.
Record the expected supporting passage for each question.
Run retrieval.
Compare retrieved chunks with the expected passages.
Repeat after changing chunking parameters.

Useful metrics include:

Recall@k
Precision@k
Mean Reciprocal Rank (MRR)
NDCG

Even a benchmark containing 100–200 realistic user queries often reveals weaknesses that casual manual testing misses.

When evaluating, change only one variable at a time.

For example:

Chunk size
Overlap
Boundary strategy
Embedding model
Reranker

Otherwise, it becomes impossible to identify which modification actually improved retrieval.

Every dataset behaves differently, but these defaults work well for many applications.

Content	Recommended strategy
Markdown documentation	Heading-based
API references	Section-based
Blogs	Paragraph-aware
PDFs	Paragraph-aware after extraction
Source code	Function or class boundaries
Meeting transcripts	Speaker turns
Logs	Fixed-size
FAQs	One question per chunk

If you’re starting a new RAG project, a strong baseline is:

Token-based chunk sizing
Structure-aware boundaries whenever available
Approximately 10–20% overlap
Complete code blocks
Retrieval evaluation before changing embedding models

Avoid these frequent pitfalls:

Splitting sentences unnecessarily
Breaking code blocks
Mixing multiple unrelated topics into one chunk
Creating chunks that are too small to answer questions independently
Creating oversized chunks containing several sections
Assuming larger overlap always improves retrieval
Optimizing embedding models before measuring retrieval quality
Evaluating only chunk counts instead of retrieval metrics

Many retrieval failures originate in chunk construction rather than the vector database itself.

Chunking is one of the highest-leverage optimizations in a RAG pipeline because it determines what the retriever can actually return.

The principles are straightforward:

One chunk should represent one coherent idea.
Prefer token-based sizing in production.
Preserve headings, paragraphs, and code boundaries whenever possible.
Use overlap sparingly instead of duplicating large portions of text.
Measure Recall@k, MRR, or NDCG with realistic queries.
Tune chunking before replacing your embedding model.

Small improvements in chunk construction often produce larger retrieval gains than changing prompts or increasing context windows.

Take one real knowledge base from your own project, implement both fixed-size and structure-aware chunking, and benchmark them using Recall@k or MRR against realistic user questions. The results will tell you far more than intuition—and they’ll help you improve retrieval quality based on evidence rather than guesswork.