Hybrid search: combining BM25 keyword and dense vectors

LEARN · RAG & AGENTIC DEVELOPMENT

Why one retrieval method is rarely enough

Dense vector retrieval is excellent at matching meaning.

A user might search for “customers are billed twice when a callback repeats,” while the relevant document says, “use an idempotency key to deduplicate payment webhooks.” The vocabulary differs, but an embedding model can place the query and document near each other because they describe the same underlying problem.

BM25 solves a different problem. It preserves literal evidence:

Error codes such as PX-417
Product identifiers such as SKU-XQ9-441
Function and class names
Version numbers
Quoted phrases
Customer or project names
Domain-specific terminology

A query containing PX-417 should strongly favor a document that actually contains PX-417, even when several other documents discuss semantically similar payment failures.

Neither method dominates every query. Dense retrieval can understand paraphrases but blur rare tokens. BM25 can preserve exact terms but miss documents expressed with different vocabulary.

A robust hybrid pipeline therefore separates four jobs:

Retrieve lexical candidates with BM25.
Retrieve semantic candidates with dense embeddings.
Fuse the two rankings.
Rerank the strongest candidates with a cross-encoder.

That pattern works for documentation search, retrieval-augmented generation, support knowledge bases, product catalogs, internal wikis, agent memory, source-code search, and enterprise discovery.

The four-stage retrieval pipeline

The stages are deliberately separate because each has a different cost and purpose.

Stage 1: BM25 lexical retrieval

BM25 scores a document from its query-term overlap while considering:

How rare each query term is across the collection
How often the term occurs in the document
The document’s length relative to the average document
Term-frequency saturation, so repeating a word many times produces diminishing returns

A readable version of the equation is:

BM25(q, d) = Σ IDF(t) × [f(t,d) × (k₁ + 1)] / [f(t,d) + k₁ × (1 − b + b × |d| / avgdl)]

The main symbols are:

f(t,d): the frequency of term t in document d
avgdl: the collection’s average document length
k₁: the term-frequency saturation control
b: the document-length normalization control

Do not begin a search project by blindly tuning k₁ and b.

The following usually matter more:

Whether titles and identifiers are indexed
How punctuation-heavy identifiers are tokenized
Whether unrelated topics are combined into one chunk
Whether boilerplate overwhelms useful content
Whether stemming helps or damages domain terms
Whether duplicate documents occupy the candidate list
Whether outdated documents remain searchable

BM25 is especially useful when the query itself contains evidence that should not be generalized away.

Stage 2: dense vector retrieval

A dense embedding model maps a query or document into a fixed-length numeric vector. Documents can then be ranked by vector similarity instead of literal term overlap.

When vectors are normalized to unit length, cosine similarity can be calculated with a dot product:

cosine_similarity(query, document) = normalized_query · normalized_document

Dense retrieval helps when:

The query paraphrases the answer.
The user describes a problem instead of naming the solution.
Synonyms differ.
The user types a full natural-language question.
The document and query share little exact vocabulary.
Search must work across multiple languages.

The current Sentence Transformers API recommends encode_query() and encode_document() for information-retrieval workloads where queries and documents play distinct roles. The methods can apply model-specific query or document behavior when the loaded model defines it. The same API supports normalized NumPy output, which lets a small local system use a matrix-vector product for exact similarity search.

Dense retrieval is not an exact-match guarantee. An embedding compresses an entire passage into a finite representation. A rare identifier may contribute less to the final vector than the passage’s broader topic.

Stage 3: reciprocal rank fusion

BM25 scores and cosine similarities should not normally be added directly.

A BM25 score of 11.8 and a cosine similarity of 0.71 do not share a universal scale. Their distributions can change with:

The query
The corpus
The embedding model
The tokenizer
The indexed fields
The BM25 implementation
The document-length distribution

Reciprocal rank fusion, usually abbreviated RRF, avoids raw-score calibration. It uses ranking positions:

RRF(d) = Σ wᵢ / (k + rankᵢ(d))

Where:

rankᵢ(d) is the document’s position in ranking i
wᵢ is an optional retriever weight
k is a constant that controls how sharply top positions are favored
A document missing from a ranking contributes nothing from that branch

Suppose a document is:

Rank 1 in BM25
Rank 5 in dense retrieval

With equal weights and k = 60, its fused score is:

1 / 61 + 1 / 65 = 0.03178

Another document that is rank 2 only in dense retrieval receives:

1 / 62 = 0.01613

The first document wins because two independent retrievers support it.

RRF is a strong default because:

It is simple.
It is deterministic.
It does not assume compatible score scales.
A document can survive by performing very well in one branch.
Agreement between branches is naturally rewarded.
Per-branch rankings remain easy to inspect.

Start with equal weights. Add weights only after measuring representative queries.

Stage 4: cross-encoder reranking

A dense retriever embeds the query and each document independently. That independence makes document embeddings reusable and candidate generation fast.

A cross-encoder evaluates a query and candidate document together. The model can directly inspect relationships between every token in the pair.

That additional interaction makes a cross-encoder useful for distinctions such as:

Whether a document actually answers the question
Whether two similar-looking error messages have different causes
Whether a negation changes the meaning
Whether a specific product identifier matches the requested accessory
Whether a general troubleshooting guide is less relevant than a precise fix

The cost is that every query-document pair requires inference. A cross-encoder should therefore rerank a shortlist, not scan an entire large corpus. Sentence Transformers documents cross-encoders as a standard second-stage reranking mechanism and provides batched predict() and rank() APIs for candidate pairs.

A practical request might:

Retrieve 50 BM25 candidates.
Retrieve 50 dense candidates.
Fuse and deduplicate them.
Rerank the strongest 20–100.
Return the best 5–20.

Those numbers are starting points. Corpus size, hardware, document length, traffic, and relevance requirements should determine the final values.

The current local stack

The runnable implementation below uses:

BM25S 0.3.10
Sentence Transformers 5.6.1
PyTorch 2.10 or newer
IBM Granite Embedding 97M Multilingual R2
Ettin Reranker 17M v1
NumPy for exact dense scoring
A plain-Python RRF implementation

BM25S 0.3.10 was released on July 22, 2026, and Sentence Transformers 5.6.1 followed on July 23, 2026. Sentence Transformers currently requires Python 3.10 or newer.

The embedding model is ibm-granite/granite-embedding-97m-multilingual-r2, released on April 29, 2026. It produces 384-dimensional vectors, supports input lengths up to 32,768 tokens, and was trained for multilingual retrieval and code retrieval. It is directly compatible with Sentence Transformers.

The reranker is cross-encoder/ettin-reranker-17m-v1, part of a Sentence Transformers reranker family released in May 2026. The models were published as native cross-encoders with released training data and recipes. The 17M variant is small enough for a practical local tutorial while still representing a current reranking architecture rather than a legacy MiniLM default.

Build a runnable local hybrid search engine

The example runs without a vector database. It is intended for learning, evaluation, unit tests, and small collections.

For a large production corpus, the same pipeline can be retained while replacing the NumPy matrix with an approximate nearest-neighbor index or managed search engine.

Project structure

hybrid-search/
├── requirements.txt
└── hybrid_search.py

Create the environment

The following commands assume a Bash-compatible shell on Linux or macOS.

mkdir hybrid-search
cd hybrid-search
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

Create requirements.txt:

bm25s==0.3.10
numpy>=2.0,<3
sentence-transformers==5.6.1
torch>=2.10,<3

Install the project:

python -m pip install -r requirements.txt

The first execution downloads the embedding model and reranker. Later executions use the local model cache unless the cache is removed or a different revision is requested.

The complete implementation

Create hybrid_search.py:

from __future__ import annotations

from dataclasses import dataclass
from typing import Iterable, Literal, Sequence

import bm25s
import numpy as np
from sentence_transformers import CrossEncoder, SentenceTransformer

Strategy = Literal["bm25", "dense", "hybrid", "hybrid_rerank"]


@dataclass(frozen=True)
class Document:
    id: str
    title: str
    text: str

    @property
    def searchable_text(self) -> str:
        return f"{self.title}\n{self.text}"


@dataclass(frozen=True)
class SearchResult:
    document: Document
    reranker_score: float
    rrf_score: float
    bm25_rank: int | None
    dense_rank: int | None


DOCUMENTS = [
    Document(
        "payments-px-417",
        "PX-417 after rotating a payment API key",
        (
            "PX-417 indicates that a checkout worker is still using a cached "
            "payment credential. Refresh the secret, restart workers that cache "
            "the old key, and verify the active key identifier before retrying."
        ),
    ),
    Document(
        "payments-key-rotation",
        "Rotate payment credentials without checkout downtime",
        (
            "Create the replacement credential first, deploy it alongside the "
            "existing credential, verify successful transactions, and revoke "
            "the previous credential after every worker has refreshed."
        ),
    ),
    Document(
        "payments-idempotency",
        "Prevent duplicate charges from repeated callbacks",
        (
            "Attach a stable idempotency key to each payment operation. Store "
            "the completed result and return it when a webhook or client retries "
            "the same operation."
        ),
    ),
    Document(
        "orders-stuck",
        "Orders remain in processing state",
        (
            "Inspect the order-event consumer, dead-letter queue, inventory "
            "reservation, and payment-completion event. Replaying an event is "
            "safe only when handlers are idempotent."
        ),
    ),
    Document(
        "catalog-sku-xq9",
        "Replacement charger for SKU-XQ9-441",
        (
            "SKU-XQ9-441 uses the 65-watt USB-C charger with model identifier "
            "PWR-65C. The older barrel connector is not compatible."
        ),
    ),
    Document(
        "catalog-charger",
        "Choose a compatible laptop charger",
        (
            "Match connector type, supported voltage, power-delivery profile, "
            "and minimum wattage. Higher supported wattage is acceptable when "
            "the device negotiates power over USB-C."
        ),
    ),
    Document(
        "returns-label",
        "A customer cannot download a return label",
        (
            "Confirm that the order is eligible, the address has a supported "
            "carrier, and the label-generation service returned a PDF. Retry "
            "only transient carrier failures."
        ),
    ),
    Document(
        "search-zero-results",
        "Reduce zero-result product searches",
        (
            "Analyze failed queries, add spelling correction, preserve catalog "
            "identifiers, support synonyms, and combine semantic retrieval with "
            "keyword matching."
        ),
    ),
]


EVALUATION_QUERIES = [
    ("PX-417 after changing payment credentials", {"payments-px-417"}),
    (
        "buyers are billed twice when a callback is delivered again",
        {"payments-idempotency"},
    ),
    ("replacement power supply for SKU-XQ9-441", {"catalog-sku-xq9"}),
    ("paid orders are trapped in processing", {"orders-stuck"}),
    ("the return shipping PDF will not generate", {"returns-label"}),
]


class HybridSearchEngine:
    def __init__(
        self,
        documents: Sequence[Document],
        embedding_model_name: str = (
            "ibm-granite/granite-embedding-97m-multilingual-r2"
        ),
        reranker_model_name: str = "cross-encoder/ettin-reranker-17m-v1",
    ) -> None:
        if not documents:
            raise ValueError("At least one document is required.")

        self.documents = list(documents)
        self.texts = [document.searchable_text for document in self.documents]

        corpus_tokens = bm25s.tokenize(self.texts, stopwords="en")
        self.bm25 = bm25s.BM25()
        self.bm25.index(corpus_tokens)

        self.embedding_model = SentenceTransformer(
            embedding_model_name,
            trust_remote_code=False,
        )
        embeddings = self.embedding_model.encode_document(
            self.texts,
            batch_size=64,
            normalize_embeddings=True,
            show_progress_bar=False,
            convert_to_numpy=True,
        )
        self.document_embeddings = np.asarray(
            embeddings,
            dtype=np.float32,
        )

        self.reranker = CrossEncoder(reranker_model_name)

    def _bm25_ids(self, query: str, limit: int) -> list[int]:
        query_tokens = bm25s.tokenize(query, stopwords="en")
        document_ids, _ = self.bm25.retrieve(query_tokens, k=limit)
        return [int(document_id) for document_id in document_ids[0]]

    def _dense_ids(self, query: str, limit: int) -> list[int]:
        query_embedding = self.embedding_model.encode_query(
            query,
            normalize_embeddings=True,
            show_progress_bar=False,
            convert_to_numpy=True,
        )
        query_vector = np.asarray(
            query_embedding,
            dtype=np.float32,
        ).reshape(-1)

        similarities = self.document_embeddings @ query_vector
        return [
            int(document_id)
            for document_id in np.argsort(-similarities)[:limit]
        ]

    @staticmethod
    def reciprocal_rank_fusion(
        rankings: Sequence[Sequence[int]],
        weights: Sequence[float] | None = None,
        rank_constant: int = 60,
    ) -> list[tuple[int, float]]:
        if rank_constant < 1:
            raise ValueError("rank_constant must be positive.")

        if weights is None:
            weights = [1.0] * len(rankings)

        if len(weights) != len(rankings):
            raise ValueError("Each ranking must have exactly one weight.")

        fused_scores: dict[int, float] = {}

        for ranking, weight in zip(rankings, weights, strict=True):
            if weight < 0:
                raise ValueError("Retriever weights cannot be negative.")

            for rank, document_id in enumerate(ranking, start=1):
                contribution = weight / (rank_constant + rank)
                fused_scores[document_id] = (
                    fused_scores.get(document_id, 0.0) + contribution
                )

        return sorted(
            fused_scores.items(),
            key=lambda item: (-item[1], item[0]),
        )

    def _rankings(
        self,
        query: str,
        *,
        candidate_k: int,
        bm25_weight: float,
        dense_weight: float,
        rank_constant: int,
    ) -> tuple[list[int], list[int], list[tuple[int, float]]]:
        bm25_ids = self._bm25_ids(query, candidate_k)
        dense_ids = self._dense_ids(query, candidate_k)
        fused = self.reciprocal_rank_fusion(
            [bm25_ids, dense_ids],
            weights=[bm25_weight, dense_weight],
            rank_constant=rank_constant,
        )
        return bm25_ids, dense_ids, fused

    def _rerank(
        self,
        query: str,
        candidate_ids: Sequence[int],
    ) -> list[tuple[int, float]]:
        pairs = [
            (query, self.documents[document_id].searchable_text)
            for document_id in candidate_ids
        ]
        scores = np.asarray(
            self.reranker.predict(
                pairs,
                batch_size=16,
                show_progress_bar=False,
                convert_to_numpy=True,
            )
        ).reshape(-1)

        return sorted(
            zip(candidate_ids, scores, strict=True),
            key=lambda item: -float(item[1]),
        )

    def rank(
        self,
        query: str,
        *,
        strategy: Strategy,
        top_k: int = 5,
        candidate_k: int = 8,
        rerank_k: int = 6,
        bm25_weight: float = 1.0,
        dense_weight: float = 1.0,
        rank_constant: int = 60,
    ) -> list[int]:
        query = query.strip()
        if not query:
            raise ValueError("Query cannot be empty.")
        if top_k < 1:
            raise ValueError("top_k must be positive.")

        collection_size = len(self.documents)
        candidate_k = min(max(candidate_k, top_k), collection_size)
        rerank_k = min(max(rerank_k, top_k), collection_size)

        bm25_ids, dense_ids, fused = self._rankings(
            query,
            candidate_k=candidate_k,
            bm25_weight=bm25_weight,
            dense_weight=dense_weight,
            rank_constant=rank_constant,
        )
        fused_ids = [document_id for document_id, _ in fused]

        if strategy == "bm25":
            return bm25_ids[:top_k]
        if strategy == "dense":
            return dense_ids[:top_k]
        if strategy == "hybrid":
            return fused_ids[:top_k]
        if strategy == "hybrid_rerank":
            reranked = self._rerank(query, fused_ids[:rerank_k])
            return [
                document_id
                for document_id, _ in reranked[:top_k]
            ]

        raise ValueError(f"Unknown strategy: {strategy}")

    def search(
        self,
        query: str,
        *,
        top_k: int = 4,
        candidate_k: int = 8,
        rerank_k: int = 6,
        bm25_weight: float = 1.0,
        dense_weight: float = 1.0,
        rank_constant: int = 60,
    ) -> list[SearchResult]:
        query = query.strip()
        if not query:
            raise ValueError("Query cannot be empty.")

        collection_size = len(self.documents)
        candidate_k = min(max(candidate_k, top_k), collection_size)
        rerank_k = min(max(rerank_k, top_k), collection_size)

        bm25_ids, dense_ids, fused = self._rankings(
            query,
            candidate_k=candidate_k,
            bm25_weight=bm25_weight,
            dense_weight=dense_weight,
            rank_constant=rank_constant,
        )
        fused = fused[:rerank_k]
        candidate_ids = [document_id for document_id, _ in fused]
        reranked = self._rerank(query, candidate_ids)

        bm25_positions = {
            document_id: rank
            for rank, document_id in enumerate(bm25_ids, start=1)
        }
        dense_positions = {
            document_id: rank
            for rank, document_id in enumerate(dense_ids, start=1)
        }
        fused_score_by_id = dict(fused)

        return [
            SearchResult(
                document=self.documents[document_id],
                reranker_score=float(score),
                rrf_score=float(fused_score_by_id[document_id]),
                bm25_rank=bm25_positions.get(document_id),
                dense_rank=dense_positions.get(document_id),
            )
            for document_id, score in reranked[:top_k]
        ]


def reciprocal_rank(
    ranked_ids: Sequence[str],
    relevant_ids: set[str],
) -> float:
    for rank, document_id in enumerate(ranked_ids, start=1):
        if document_id in relevant_ids:
            return 1.0 / rank
    return 0.0


def recall_at_k(
    ranked_ids: Sequence[str],
    relevant_ids: set[str],
    k: int,
) -> float:
    if not relevant_ids:
        return 0.0

    retrieved = set(ranked_ids[:k])
    return len(retrieved & relevant_ids) / len(relevant_ids)


def evaluate(engine: HybridSearchEngine, k: int = 5) -> None:
    strategies: list[Strategy] = [
        "bm25",
        "dense",
        "hybrid",
        "hybrid_rerank",
    ]

    print(f"\nOFFLINE EVALUATION AT K={k}")
    print("strategy          MRR      recall")

    for strategy in strategies:
        reciprocal_ranks: list[float] = []
        recalls: list[float] = []

        for query, relevant_ids in EVALUATION_QUERIES:
            integer_ids = engine.rank(
                query,
                strategy=strategy,
                top_k=k,
            )
            ranked_ids = [
                engine.documents[document_id].id
                for document_id in integer_ids
            ]
            reciprocal_ranks.append(
                reciprocal_rank(ranked_ids, relevant_ids)
            )
            recalls.append(
                recall_at_k(ranked_ids, relevant_ids, k)
            )

        print(
            f"{strategy:<17}"
            f"{float(np.mean(reciprocal_ranks)):.4f}   "
            f"{float(np.mean(recalls)):.4f}"
        )


def print_results(
    query: str,
    results: Iterable[SearchResult],
) -> None:
    print(f"\nQUERY: {query}")

    for rank, result in enumerate(results, start=1):
        bm25_rank = result.bm25_rank or "-"
        dense_rank = result.dense_rank or "-"

        print(
            f"{rank}. {result.document.id} | "
            f"reranker={result.reranker_score:.4f} | "
            f"rrf={result.rrf_score:.5f} | "
            f"bm25_rank={bm25_rank} | "
            f"dense_rank={dense_rank}"
        )
        print(f"   {result.document.title}")


def main() -> None:
    engine = HybridSearchEngine(DOCUMENTS)
    evaluate(engine)

    queries = [
        "PX-417 appeared after we rotated the checkout secret",
        "customers get charged twice when a payment callback repeats",
        "which power adapter works with XQ9 441",
        "orders never leave processing after payment succeeds",
    ]

    for query in queries:
        print_results(query, engine.search(query))


if __name__ == "__main__":
    main()

Run the program:

python hybrid_search.py

The exact scores and, occasionally, the ordering of close candidates can vary across hardware, PyTorch builds, and model revisions. Treat the diagnostic ranks and overall behavior as the important output rather than expecting fixed floating-point values.

Walking through the implementation

The same documents feed both retrieval branches

The example creates searchable_text by joining each document’s title and body.

That representation is sent to both:

The BM25 tokenizer and index
The dense embedding model

Using the same source text makes the demonstration easier to reason about. Production systems often use branch-specific representations.

A more mature design might:

Give title matches a larger lexical boost.
Store identifiers in a dedicated keyword field.
Embed a cleaned passage without repetitive navigation text.
Send title, passage, and selected metadata to the reranker.
Keep access-control and filtering metadata outside the text.
Group chunks under a stable parent-document identifier.

Do not embed raw application objects without inspection. Large JSON payloads can contain timestamps, tracking parameters, repeated field names, internal IDs, and unrelated metadata that weaken the useful semantic signal.

BM25S returns document positions

BM25S separates the strings being tokenized from the values returned during retrieval. When no external corpus is supplied to retrieve(), the API returns document positions and scores. Those positions map directly to the DOCUMENTS list, which makes fusion with dense retrieval straightforward.

The local implementation preserves those integer positions until the final presentation layer.

In production, replace list positions with stable document or chunk identifiers. A list index is safe only while the in-memory corpus remains immutable.

Document embeddings are computed once

The constructor encodes every document and stores the resulting matrix.

Because normalize_embeddings=True is used, each row is a unit-length vector. At query time, the engine normalizes the query and calculates all similarities with one operation:

document_embeddings @ query_vector

For a small collection, exact scoring is valuable:

It is deterministic.
It has no approximate-index recall loss.
It is easy to debug.
It provides a useful reference implementation.
It helps verify a later approximate index.

At large scale, exact matrix scoring becomes too expensive. Replace it with an approximate nearest-neighbor index, but keep a small exact test corpus for regression tests.

The two rankings remain independent

The engine does not attempt to normalize BM25 and dense scores into a shared range.

It keeps ordered document IDs such as:

BM25 ranking: document 0, document 1, document 4
Dense ranking: document 1, document 0, document 2

RRF consumes the positions.

A document appearing in both lists receives two contributions. A document appearing in only one list can still survive if that branch ranks it highly enough.

This is an important property. Hybrid retrieval should preserve complementary evidence, not require every good document to satisfy both retrievers.

The reranker sees only fused candidates

After fusion, the example sends the strongest candidates to the cross-encoder as query-document pairs.

For the duplicate-charge query, a pair looks conceptually like this:

Query: “customers get charged twice when a payment callback repeats”
Document: “Prevent duplicate charges from repeated callbacks … Attach a stable idempotency key …”

The cross-encoder reads both pieces of text together. It can recognize that idempotency directly addresses repeated payment execution even when the user never types the formal term.

Do not interpret a reranker score as a universal probability of relevance.

The score is useful primarily for ordering candidates generated for the same query. A production threshold should be selected from labeled validation data and tested against no-answer queries.

Diagnostics are part of the result object

Each displayed result includes:

Its final reranker score
Its RRF score
Its BM25 rank
Its dense rank

Those fields answer questions such as:

Did an exact identifier rescue the result?
Did only the dense branch find the paraphrase?
Did both retrievers agree?
Did the reranker reverse the fused order?
Was a relevant document absent from one branch?
Is candidate depth too shallow?

Without these diagnostics, hybrid search can become an opaque sequence of models and constants.

Cherry on the cake: the exact error code that vanished

Consider a composite support-search incident based on a common production failure pattern.

An engineer searches:

PX-417 appeared after we rotated the checkout secret

The knowledge base contains a short article titled:

PX-417 after rotating a payment API key

It also contains longer articles about:

Credential rotation
Secret deployment
Checkout outages
Cached configuration
Worker restarts
Payment-provider authentication

A pure-vector search sees strong semantic similarity across all of those documents. The general key-rotation guide may rank first because it discusses the broad topic in more detail. An outage runbook may rank second because it contains multiple concepts from the query.

The embarrassing result is that the short article containing the literal code PX-417 appears several positions lower—or falls outside the candidate window entirely.

Nothing is “broken” in the embedding model. The vector is doing what it was trained to do: represent the passage’s overall meaning. The identifier occupies only a small part of that meaning.

BM25 behaves differently. It sees a rare token sequence and strongly favors the document containing it. The hybrid pipeline keeps the exact-code article in the candidate pool. The reranker can then verify that its cached-credential explanation matches the complete question.

The lesson is simple:

Semantic plausibility is not the same as evidential precision.

A document can discuss the right topic without containing the exact evidence needed by the user.

Evaluate retrieval before tuning it

The included script compares four strategies:

BM25 only
Dense retrieval only
RRF without reranking
RRF followed by cross-encoder reranking

It reports:

Mean reciprocal rank
Recall at k

Mean reciprocal rank

Reciprocal rank rewards placing the first relevant result near the top.

If the first relevant result appears at:

Rank 1, the reciprocal rank is 1.0
Rank 2, it is 0.5
Rank 5, it is 0.2
No retrieved rank, it is 0.0

MRR averages that value across the evaluation queries.

MRR is useful for navigational and question-answering search, where users often need one strong result quickly.

Recall at k

Recall at k asks how many known relevant documents were preserved in the first k results.

If a query has four relevant documents and three appear in the first ten results:

Recall@10 = 3 / 4 = 0.75

Recall is especially important before reranking. A reranker cannot recover a document that both candidate generators discarded.

Why the included evaluation is only a smoke test

Five queries are enough to demonstrate the machinery, not to validate a production search engine.

A useful relevance set should include:

Exact identifiers
Natural-language paraphrases
Misspellings
Abbreviations
Long conversational questions
Short ambiguous queries
Multiple acceptable answers
No-answer queries
Recently updated content
Permission-sensitive content
Queries from every supported language
Queries from major user segments

Where possible, collect hundreds of representative judgments.

Split them into:

A development set for tuning
A held-out test set for final comparison

Do not repeatedly optimize against the test set. That merely turns the test set into another development set.

Tune the system in the right order

1. Fix the corpus before changing weights

Inspect failed queries for:

Missing titles
Broken document extraction
Duplicate chunks
Boilerplate-heavy content
Outdated documents
Extremely long chunks
Tiny context-free fragments
Missing identifiers
Multiple unrelated topics in one chunk
Incorrect permissions
Poor language detection

A ranking model cannot reconstruct content that was never indexed.

2. Measure branch-level candidate recall

Evaluate BM25 and dense retrieval separately.

For every relevant document, record:

Its BM25 rank
Its dense rank
Whether either branch found it
Whether it survived fusion
Whether the reranker improved or damaged its position

This separates retrieval failures from reranking failures.

A weak final result can have several different causes:

Both candidate generators missed the document.
One branch found it, but fusion depth removed it.
Fusion preserved it, but rerank depth excluded it.
The reranker scored it incorrectly.
A filter removed it.
The indexed passage lacked enough context.

Each failure requires a different fix.

3. Increase candidate depth until recall plateaus

If relevant documents often appear around BM25 rank 40 or dense rank 35, retrieving ten candidates from each branch guarantees avoidable misses.

Increase candidate_k gradually and plot recall against latency.

Do not immediately retrieve thousands of candidates. That can:

Increase memory use
Transfer too much data
Produce unstable tail latency
Overload the reranker
Fill the candidate pool with duplicates
Hide problems in document preparation

The goal is not maximum candidate count. It is sufficient recall within an acceptable budget.

4. Tune the RRF constant

A smaller RRF constant creates a larger difference between early and later ranks.

A larger constant makes their contributions more similar.

Test several values against the development set. Keep the value that improves the target metric without harming important query classes.

Do not treat 60 as a mathematical law. It is a sensible starting point.

5. Tune lexical and dense weights with evidence

Equal weights are the correct default when no relevance data exists.

After collecting judgments, test alternatives.

An identifier-heavy support corpus may benefit from stronger lexical influence. A conversational discovery experience may benefit from stronger semantic influence.

Query-aware routing can be even better:

Detected error code: increase lexical weight.
Detected SKU or UUID: increase lexical weight.
Quoted phrase: prioritize exact matching.
Full natural-language question: retain strong dense retrieval.
Broad category query: combine retrieval with popularity or business signals.
Navigational title query: boost canonical titles.

Keep routing rules inspectable. A complicated learned router can create another opaque ranking layer.

6. Tune rerank depth separately

Reranking more candidates may improve final quality, but every additional pair adds inference cost.

Measure:

Median latency
p95 latency
p99 latency
Candidate pairs per request
Batch utilization
CPU or GPU utilization
Input-token distribution
Truncation frequency
Queueing time
Memory use

The fastest batch size depends on hardware, sequence lengths, model size, and concurrency. Benchmark realistic traffic rather than one isolated query.

Chunking is part of ranking

Long model context does not mean every document should be embedded as one giant passage.

A long embedding can mix several topics into one vector. The result may be broadly related to many queries but precisely relevant to none.

A practical chunk should usually contain:

A meaningful heading
Enough local context to answer a question
The important identifiers
The relevant procedure or explanation
A reference to its parent document

Chunk boundaries should follow structure when possible:

Markdown headings
HTML sections
Documentation pages
Product descriptions
Ticket messages
Code functions or classes
Log-event groups

Avoid arbitrary character windows unless the source lacks usable structure.

For final presentation, consider retrieving chunks but returning their parent documents. This prevents several neighboring chunks from the same source from occupying the entire results page.

Production architecture

A production request path commonly looks like this:

User query
    │
    ├── Query normalization
    │     ├── Language detection
    │     ├── Identifier detection
    │     ├── Spelling handling
    │     └── Permission context
    │
    ├── BM25 candidate retrieval
    │
    ├── Dense candidate retrieval
    │
    ├── Metadata and access filters
    │
    ├── Reciprocal rank fusion
    │
    ├── Cross-encoder reranking
    │
    ├── Deduplication and grouping
    │
    ├── Optional business rules
    │
    └── Final results or grounded context

The components can live in one search engine or several services.

Application-side fusion is useful when:

Lexical and vector indexes live in different systems.
You need custom experimental logic.
You require detailed per-branch diagnostics.
Different teams own different retrievers.
Query routing changes weights dynamically.

Server-side fusion is attractive when:

Both branches share one engine.
Filters must be applied consistently.
Network transfer of intermediate candidates is expensive.
Operational simplicity matters more than custom logic.

Apply permissions in every retrieval branch

Authorization is not a post-processing preference.

A semantically relevant document is invalid when the user cannot access it.

Apply tenant, user, role, project, region, and document-level restrictions to both lexical and dense retrieval.

Filtering only after taking a tiny global top-k can damage recall. Unauthorized documents may consume candidate slots before the allowed results are considered.

The safer pattern is:

Establish the user’s access context.
Apply compatible filters during each retrieval branch.
Fuse only eligible candidates.
Recheck authorization before returning content.

Treat search-result snippets, generated answers, citations, cached results, and analytics as part of the same security boundary.

Keep stable identifiers

Fusion depends on recognizing that two retrievers returned the same item.

Use stable IDs across:

Source documents
Chunks
BM25 records
Vector records
Reranker inputs
Analytics events
User feedback
Cache keys

A practical identity scheme might use:

document_id: stable source identity
chunk_id: stable chunk identity within a document
content_version: source revision
index_version: retrieval build
embedding_version: model and preprocessing version

Do not use an array position as a persistent production ID.

Version models, chunks, and indexes

Embedding vectors from different models or preprocessing pipelines should not be mixed silently.

Record metadata such as:

{
  "document_id": "payments-px-417",
  "chunk_id": "payments-px-417:0001",
  "embedding_model": "ibm-granite/granite-embedding-97m-multilingual-r2",
  "embedding_revision": "approved-model-commit",
  "reranker_model": "cross-encoder/ettin-reranker-17m-v1",
  "chunker_version": "2026-08-04",
  "content_checksum": "sha256:replace-with-real-checksum",
  "indexed_at": "2026-08-04T12:00:00Z"
}

When any of the following changes, build a new index version:

Embedding model
Model revision
Tokenizer
Text normalization
Chunking strategy
Title formatting
Metadata inclusion
Language routing
Permission representation

Deploy the new index alongside the previous one. Run validation queries, compare metrics, and keep a rollback path.

Security note: model files are executable-risk inputs

Model loading belongs in your software supply-chain threat model.

A 2026 PyTorch advisory, CVE-2026-24747, described a malicious-checkpoint path that could potentially lead to arbitrary code execution even when torch.load(..., weights_only=True) was used. The advisory affected PyTorch versions through 2.9.1 and lists 2.10.0 and later as patched. That is why the example requires torch>=2.10.

Practical safeguards include:

Keep PyTorch and model-loading libraries patched.
Prefer models distributed with safetensors.
Download only from reviewed publishers or controlled storage.
Pin approved model revisions.
Record artifact hashes.
Scan dependencies in CI.
Avoid loading arbitrary user-supplied checkpoints.
Keep trust_remote_code=False unless custom repository code has been reviewed.
Run model services with restricted filesystem and network permissions.
Rebuild indexes after a model artifact changes.
Treat model caches as production artifacts, not disposable downloads.

A model repository is a dependency source. Apply the same discipline used for container images and application packages.

Log enough to explain rankings

For sampled or privacy-approved queries, capture:

Query classification
Applied filters
BM25 candidates and ranks
Dense candidates and ranks
RRF contributions
Reranker scores and order
Model revisions
Index version
Latency by stage
Click, reformulation, or resolution outcome

Do not automatically store every raw query.

Search queries can contain:

Access tokens
Email addresses
Customer names
Internal project names
Confidential incident details
Personal information
Proprietary source-code fragments

Redact or hash sensitive fields, define retention periods, and restrict access to ranking logs.

Cache the right artifacts

Useful cache targets include:

Document embeddings
Tokenized lexical indexes
Repeated query embeddings
Reranker outputs for identical query-document-version pairs
Final results for safe, non-personalized queries

Cache keys must include the versions that affect correctness:

Query normalization version
Embedding model and revision
Reranker model and revision
Index version
Content version
Permission context
Filter set
Fusion configuration

A cached result generated before a permission or content change can be worse than a cache miss.

Know when hybrid retrieval is not enough

Hybrid candidate generation is a foundation, not the entire ranking product.

You may still need:

Date and price filters
Geographic constraints
Freshness decay
Popularity signals
Inventory availability
Diversity constraints
Parent-document grouping
Deduplication
Spelling correction
Query rewriting
Multilingual routing
Domain-specific fine-tuning
Curated navigational boosts
A no-answer threshold
Policy and compliance checks

For retrieval-augmented generation, also verify that the generated response is grounded in the selected passages. Better retrieval reduces hallucination risk but does not eliminate it.

A practical launch checklist

Before shipping, verify that:

BM25 and dense retrieval are evaluated independently.
Hybrid fusion improves or safely matches the strongest individual branch.
The reranker improves held-out ranking quality.
Exact identifiers remain searchable.
Natural-language paraphrases remain searchable.
Candidate recall is measured before reranking.
Permission filters apply to every branch.
Stable IDs are shared across indexes.
Model revisions are recorded.
Reindexing is repeatable.
Model artifacts come from approved sources.
PyTorch is newer than the affected CVE range.
Latency is measured at realistic concurrency.
Long documents are chunked deliberately.
Duplicate content is controlled.
No-answer queries appear in evaluation.
Ranking diagnostics are observable.
Sensitive queries are protected in logs.
The previous index can be restored quickly.

Put the pattern into practice

Run the local project, replace the sample documents with a neutral slice of your own catalog or knowledge base, and label at least 50 real queries before changing the weights.

Compare these four configurations:

BM25 only
Dense retrieval only
Equal-weight RRF
RRF followed by cross-encoder reranking

Measure candidate recall before reranking and ranking quality after reranking. Then inspect every query where the systems disagree.

Those disagreements reveal what your users actually need:

Exact identifiers
Paraphrases
Misspellings
Better chunks
Fresher content
More context
Stronger filtering
Different candidate depths

Build the evaluation set first, preserve both lexical and semantic evidence, and require every ranking change to prove that it improves the searches your users really perform.