Build a minimal RAG pipeline: chunk, embed, retrieve, answer

LEARN · RAG & AGENTIC DEVELOPMENT

Retrieval-Augmented Generation (RAG) is one of the most practical patterns for building AI applications that work with private or constantly changing information.

Instead of asking a language model to remember everything, a RAG pipeline retrieves relevant information at query time and gives that context to the model before generating an answer.

A minimal RAG system has four core stages:

Chunking: split documents into useful pieces.
Embeddings: convert text into vectors that capture semantic meaning.
VectorDB retrieval: search those vectors to find relevant context.
Answer generation: use the retrieved context to produce a response.

This architecture powers many real-world systems:

Internal company knowledge assistants.
Documentation search tools.
Customer support copilots.
Research assistants.
Enterprise AI applications.

The key idea is simple:

Better retrieved context usually leads to better answers.

RAG is not a replacement for good data engineering. It is a search and generation pipeline where every step affects the final result.

The RAG pipeline explained

Imagine a company handbook stored as a PDF.

A user asks:

“How many vacation days do employees receive after five years?”

A traditional keyword search may look for exact terms like:

vacation
five years
employees

An embedding-based search can understand related concepts:

annual leave entitlement
time off policy
years of service benefits

The basic workflow:

Load documents.
Split documents into chunks.
Create embeddings for each chunk.
Store vectors in a VectorDB.
Embed the user’s question.
Retrieve similar chunks.
Send the retrieved context to a language model.

The language model does not automatically know your documents. It only sees the information provided in the prompt.

Step 1: Chunk documents carefully

Chunking is one of the most important parts of RAG.

Large documents usually cannot be sent directly to a model because of context limits, cost, and unnecessary information. Splitting documents into smaller sections makes retrieval faster and more focused.

A basic chunking strategy:

Choose a target chunk size.
Add overlap between chunks.
Keep related information together.
Preserve document structure when possible.

For example, a product manual might be better split by headings:

Installation

Requirements:
- Operating system versions
- Hardware requirements

Setup:
- Configuration steps
- First launch instructions

A legal contract may require different handling:

Keep clauses together.
Preserve section numbers.
Avoid splitting definitions from the rules they explain.

A common beginner mistake is choosing a chunk size without testing retrieval quality.

There is no universal best chunk size.

Good chunking depends on:

Document type.
Question style.
Retrieval method.
Model context window.
Evaluation results.

Useful approaches include:

Splitting documentation by headings.
Splitting articles by paragraphs.
Splitting source code by functions or classes.
Keeping tables and lists intact.

Step 2: Create Embeddings

Embeddings convert text into numerical vectors.

For example:

"How do I reset my password?"

↓

[0.021, -0.184, 0.552, ...]

The exact numbers are not meaningful by themselves. Their purpose is to represent relationships between pieces of text.

Texts with similar meanings tend to have vectors that are close together in the embedding space.

Modern embedding options include:

OpenAI text-embedding-3-small.
OpenAI text-embedding-3-large.
Open-source models such as BGE and E5 families.

The best embedding model is not always the largest one. Retrieval quality depends on factors such as:

Your language.
Your domain.
Your document style.
Your evaluation benchmark.
Your search strategy.

For example, a small model can perform very well for a narrow internal knowledge base, while a different model may work better for multilingual documents.

One important rule:

Store document embeddings and query embeddings using compatible embedding models.

If you change embedding models, you usually need to rebuild your vector index.

Step 3: Store vectors in a VectorDB

A VectorDB stores embeddings and makes similarity search possible.

Common choices include:

Managed vector databases.
Open-source vector databases.
PostgreSQL databases with vector extensions.

A stored vector record usually contains:

The original text chunk.
The embedding vector.
Metadata.

Example metadata:

{
  "document": "employee_handbook.pdf",
  "section": "vacation_policy",
  "page": 12
}

Metadata enables important production features:

Filtering by document type.
Restricting results by user permissions.
Searching only recent versions.
Tracking document sources.

A production RAG system should treat metadata and access control as first-class features.

Build a minimal RAG pipeline in Python

The following example shows the complete RAG flow:

Create embeddings.
Store vectors in memory.
Retrieve the closest match.
Generate an answer.

It intentionally avoids a framework so each step is visible.

from openai import OpenAI
import numpy as np

client = OpenAI()

documents = [
    "Employees receive 20 vacation days per year after completing probation.",
    "Password resets can be requested through the security portal.",
    "Remote work is available three days per week with manager approval.",
]


def create_embeddings(texts):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
    )

    return [item.embedding for item in response.data]


def cosine_similarity(vector_a, vector_b):
    return np.dot(vector_a, vector_b) / (
        np.linalg.norm(vector_a) * np.linalg.norm(vector_b)
    )


document_vectors = create_embeddings(documents)

question = "How much vacation time do employees get?"

question_vector = create_embeddings([question])[0]

scores = [
    cosine_similarity(question_vector, vector)
    for vector in document_vectors
]

best_index = int(np.argmax(scores))
context = documents[best_index]

response = client.responses.create(
    model="gpt-5-mini",
    input=(
        "Answer using this context:\n"
        f"{context}\n\n"
        f"Question: {question}"
    ),
)

print(response.output_text)

This small example demonstrates the core pattern.

A production system would add:

A real VectorDB.
Document ingestion pipelines.
Authentication.
Permission-aware retrieval.
Monitoring.
Automated evaluations.
Better retrieval strategies.

The foundation remains the same:

chunk → embed → retrieve → answer

Improving retrieval quality

A RAG application is limited by the quality of the context it retrieves.

Several improvements can make a large difference.

Add metadata filtering

Instead of searching everything:

Search only documents a user can access.
Search only approved versions.
Search only relevant departments.

This improves both quality and security.

Use hybrid search

Vector search is excellent for meaning.

Keyword search is excellent for exact matches.

Examples where keyword search helps:

Product IDs.
Error codes.
Legal references.
Version numbers.

Many production systems combine:

Semantic vector retrieval.
Traditional keyword search.

Add reranking

A common architecture:

Retrieve 20 possible chunks.
Use a reranking model to score them.
Send the best few chunks to the language model.

This helps when many documents contain similar information.

The chunking trap: smaller is not always better

A surprising RAG lesson is that making chunks smaller can reduce answer quality.

Many early enterprise RAG experiments used simple rules like:

Split every 500 characters.
Create embeddings.
Retrieve the closest text.

The problem:

A system might retrieve a sentence mentioning a product feature but miss the next paragraph containing an important limitation or exception.

The result:

The answer sounds confident.
The retrieved context is incomplete.
The final response is misleading.

The fix is often better document handling, not a bigger model.

Better approaches:

Preserve headings.
Keep related paragraphs together.
Include surrounding context.
Test multiple chunking strategies.
Measure retrieval accuracy with real questions.

RAG quality is often a data preparation problem before it is a model problem.

Cherry on the cake: CVE-2024-3094 and the hidden supply-chain risk

A useful security lesson comes from the 2024 discovery of XZ Utils backdoor.

Attackers introduced malicious code into XZ Utils, a widely used open-source compression project. The incident showed how a trusted dependency can become a security risk through a sophisticated software supply-chain attack.

Although CVE-2024-3094 was not a RAG vulnerability, the lesson applies directly to AI systems.

A RAG application may depend on many components:

PDF parsers.
Document loaders.
Open-source libraries.
Vector databases.
Authentication services.
External APIs.

A vulnerable document parser, for example, could affect a RAG pipeline before the AI model ever sees the data.

Security reviews should cover the entire AI application stack, not only the language model.

Common beginner mistakes

Avoid these early problems:

Using huge chunks: retrieval becomes noisy because each result contains too much unrelated information.
Using tiny chunks: important relationships between sentences disappear.
Skipping evaluation: a demo working once does not prove reliability.
Ignoring permissions: users may retrieve information they should not see.
Changing embedding models without re-indexing: old vectors may no longer match new query vectors.
Trusting generated answers without checking sources: retrieval quality still needs validation.

A practical production roadmap

A sensible development path:

Build a simple chunk → embed → retrieve → answer prototype.
Connect a real VectorDB.
Add metadata and permissions.
Create a question set with expected answers.
Measure retrieval quality.
Add reranking or hybrid search.
Monitor failures in production.
Improve documents and chunking based on real usage.

The basic concept stays straightforward:

RAG gives a model the right information at the right time.

The engineering challenge is making sure the retrieved information is accurate, secure, and useful.

Start building your first RAG application

Pick a small document collection, build the minimal pipeline, and measure where it fails.

Experiment with chunking, create Embeddings, connect a VectorDB, and improve one stage at a time until your AI application can reliably answer questions from your own data.

Start with the fundamentals today: chunk, embed, retrieve, answer.