Model cascades and routing: answer cheap when you can, expensive when you must

Why model cascades matter

Large language models have changed how software answers questions, writes content, summarizes documents, and automates workflows. But production AI systems rarely have a single “best” model choice for every request.

A customer asking for an order status update, an employee asking for a document summary, and an engineer debugging a distributed system failure may all interact with the same AI product. Their requests have very different complexity, risk, and value.

Sending every request to the largest available model is usually inefficient. It increases inference cost, adds latency, and can make scaling harder. More importantly, it ignores an important engineering reality: many AI requests are easy.

A well-designed AI application should answer cheap when it can and answer expensive when it must.

That is the foundation of model cascades and routing.

A model cascade is an architecture where requests move through multiple model tiers:

• A smaller, faster model handles straightforward requests.

• A routing layer evaluates whether the result is likely acceptable.

• A larger model handles difficult, ambiguous, or high-risk requests.

• Feedback from production improves future routing decisions.

This approach is useful for:

• Customer support assistants

• Internal company knowledge tools

• Coding assistants

• Document extraction systems

• AI search applications

• Content generation platforms

• Agent workflows that combine models and tools

The goal is not to avoid powerful models. The goal is to use expensive intelligence only when it creates measurable value.

The architecture of a model cascade

A production cascade usually contains four major components:

1. Request classification or routing

2. A low-cost first-pass model

3. Quality and confidence evaluation

4. Escalation to a more capable model

A simple flow looks like this:

User request
     |
     v
Routing layer
     |
     v
Small model
     |
     v
Confidence and quality checks
     |
     +----------------+
     |                |
     v                v
Return answer     Large model
                  fallback

The router does not need perfect intelligence. It only needs to make better decisions than always selecting the most expensive model.

Consider a retail assistant.

A customer asks:

“Where is my order?”

This request likely requires retrieving order information and formatting a simple response. A smaller model can often handle it.

Another customer asks:

“Compare my refund dispute against company policy, identify possible exceptions, explain relevant consumer rules, and draft a response that reduces legal risk.”

That request requires more reasoning, stronger instruction following, and possibly human review. Escalation is appropriate.

The cascade architecture turns model selection from a static configuration decision into a dynamic optimization problem.

The economics behind model routing

The main reason teams build cascades is cost efficiency, but cost is only one part of the equation.

A useful production system balances:

• Cost per request

• Response latency

• Answer quality

• Reliability

• User satisfaction

• Operational complexity

Imagine a support application receiving one million requests per month.

If every request uses a premium reasoning model, the system may deliver excellent quality but unnecessary expense.

If 85% of requests can be handled by a smaller model while maintaining customer satisfaction, the savings can be substantial.

The ideal cascade does not minimize model usage. It maximizes value per model call.

A useful mental model is:

Expected value = Quality benefit - Model cost - Failure cost

A larger model is justified when the quality improvement is worth the additional cost.

Confidence-based escalation

The most common routing strategy uses confidence thresholds.

The small model generates an answer. The system evaluates whether that answer is likely good enough.

A simple policy:

if confidence >= threshold:
    return small_model_answer
else:
    return large_model_answer

However, production systems should avoid relying only on a model saying “I am confident.”

Language models can produce fluent but incorrect answers. Confidence must be supported by additional signals.

Better routing signals include:

• Retrieval quality from a knowledge base

• Whether required fields were extracted successfully

• Structured output validation

• Agreement between multiple attempts

• Historical accuracy for similar requests

• User feedback patterns

• Request complexity classification

• Business risk level

For example, a product description generator may tolerate occasional wording mistakes. A system generating financial recommendations should have a much stricter escalation policy.

The important question is not:

“Does the model sound confident?”

The important question is:

“What happens if this answer is wrong?”

A runnable Python cascade using the OpenAI Responses API

The following example uses the current OpenAI Python SDK style with the Responses API. Install the SDK with:

pip install --upgrade openai

The example uses:

• gpt-5-mini as the inexpensive first-pass model

• gpt-5 as the stronger fallback model

• Structured JSON output

• Validation before accepting the smaller model response

import json
from openai import OpenAI

client = OpenAI()

SMALL_MODEL = "gpt-5-mini"
LARGE_MODEL = "gpt-5"


def ask_model(model, question):
    response = client.responses.create(
        model=model,
        input=[
            {
                "role": "system",
                "content": (
                    "Answer the user question. "
                    "Return only JSON with two keys: "
                    "answer and confidence. "
                    "confidence must be a number between 0 and 1."
                ),
            },
            {
                "role": "user",
                "content": question,
            },
        ],
    )

    try:
        result = json.loads(response.output_text)
    except json.JSONDecodeError:
        return {
            "answer": response.output_text,
            "confidence": 0.0,
        }

    if not isinstance(result.get("confidence"), (int, float)):
        result["confidence"] = 0.0

    return result


def cascade(question, threshold=0.85):
    small_result = ask_model(SMALL_MODEL, question)

    if small_result["confidence"] >= threshold:
        return {
            "model": SMALL_MODEL,
            "answer": small_result["answer"],
        }

    large_result = ask_model(LARGE_MODEL, question)

    return {
        "model": LARGE_MODEL,
        "answer": large_result["answer"],
    }


result = cascade(
    "Explain the warranty period for a customer who purchased a laptop."
)

print(result["model"])
print(result["answer"])

A production implementation should add:

• Request tracing

• Retry handling

• Rate-limit management

• Evaluation logging

• Privacy controls

• Human escalation paths

The architectural lesson remains simple: the expensive model should be a deliberate choice, not the default path.

Designing better escalation rules

A weak cascade asks:

“Is the small model confident?”

A stronger cascade asks:

“Given this request type, what is the expected cost of failure?”

That difference changes how routing should be designed.

Low-risk requests:

• Summarizing meeting notes

• Extracting product names

• Reformatting text

• Classifying support tickets

can use aggressive small-model routing.

High-risk requests:

• Security recommendations

• Legal interpretation

• Financial decisions

• Customer commitments

should escalate earlier.

A practical system often combines model confidence with risk scoring.

Example:

routing_score =
    model_confidence
    + retrieval_quality
    + validation_score
    - risk_penalty

A security incident assistant might require near-perfect confidence before returning a small-model answer. A personal productivity assistant might accept a lower threshold.

Measuring whether your cascade works

A cascade is successful only when it improves real business outcomes.

Track four categories of metrics.

Cost metrics

Measure:

• Average cost per request

• Percentage of requests handled by each model

• Token usage by model tier

• Monthly inference spending

A successful cascade usually increases the percentage of requests completed by the cheaper model without reducing quality.

Quality metrics

Measure:

• Human evaluation scores

• User satisfaction

• Correction frequency

• Escalation success rate

• Task completion rate

A cheaper answer is not a success if users need to repeat the request.

Latency metrics

Measure:

• Time to first token

• Total response time

• Queue delays

• Timeout frequency

Smaller models often improve user experience because they produce answers faster.

Routing metrics

The router itself needs evaluation.

Important measurements:

• False escalation: the large model was used unnecessarily

• Missed escalation: the small model answered when it should not have

Missed escalations are usually more damaging because they directly affect trust.

Building an evaluation dataset

Many teams build sophisticated routing logic but fail to build a realistic evaluation set.

A good routing dataset should contain examples of:

• Requests the small model handles successfully

• Requests requiring deeper reasoning

• Requests needing external tools

• Requests requiring human review

For a retail assistant, a neutral evaluation set might look like this:

[
  {
    "question": "Where is my order?",
    "expected_route": "small"
  },
  {
    "question": "Compare two subscription plans and recommend the better option for a business.",
    "expected_route": "large"
  },
  {
    "question": "Extract product names from this invoice.",
    "expected_route": "small"
  }
]

Synthetic examples are useful when starting. Real production traffic is usually where the most valuable routing insights appear.

Unexpected user behavior often reveals that a request category you considered simple is actually complex.

The cherry on the cake: Microsoft’s Bing Chat model-routing story

A notable real-world example of model routing economics comes from Microsoft’s early Bing Chat deployment.

When Bing Chat launched in 2023, Microsoft and OpenAI operated a system built around GPT-4 technology, but public reporting highlighted an important production reality: serving advanced models at large scale required careful management of latency, availability, and cost. Microsoft later introduced smaller and faster AI models into Copilot experiences, including models optimized for everyday interactions, while reserving more capable models for harder tasks.

The surprising fact is that large AI products increasingly behave less like “one chatbot powered by one model” and more like intelligent traffic systems. The user sees one assistant, but behind the scenes the application may select different models depending on the request.

This is the same principle behind modern model cascades: the best AI product is often not the one with the biggest model everywhere. It is the one that knows when a smaller model is enough.

A second lesson comes from security. In 2023, researchers demonstrated prompt injection risks against systems connected to external tools and data sources. The broader lesson for routing systems is that a router is part of the security boundary. Attackers may attempt to manipulate classification, force expensive processing, or trick weaker models into handling tasks they should not handle.

A secure cascade needs:

• Input filtering

• Abuse detection

• Output validation

• Logging

• Rate limits

• Clear escalation policies

Optimization and security cannot be separated.

Beyond two-model cascades

A basic cascade uses two models, but advanced systems often use multiple specialized routes.

Example:

                 User request
                      |
                      v
                Intent router
                      |
        +-------------+-------------+
        |             |             |
        v             v             v
   Small model   Specialist     Large model
                 model

Possible routes include:

• Coding-focused models for programming questions

• Retrieval-focused systems for company documents

• Vision models for image understanding

• Reasoning models for complex analysis

This application-level routing resembles a mixture-of-experts approach, where different capabilities are selected for different tasks.

The router becomes a key intelligence layer.

Common mistakes

Mistake 1: Escalating after failure

If the cost of being wrong is high, do not wait for a failed answer.

Use risk-aware routing before generation.

Mistake 2: Optimizing only for price

The cheapest response is not always the best response.

Measure customer outcomes.

Mistake 3: Hardcoding every route

Rules are useful, but user behavior changes.

Review routing decisions regularly.

Mistake 4: Ignoring operational complexity

Cascades introduce:

• More monitoring

• More evaluation work

• More deployment paths

• More failure modes

The savings must justify the architecture.

A practical rollout plan

Teams building their first production cascade should start small.

A sensible sequence:

1. Choose two models.

2. Collect representative requests.

3. Measure which tasks actually need the larger model.

4. Add confidence and validation signals.

5. Compare cost and quality before and after routing.

6. Improve thresholds continuously.

Do not begin with a complex multi-agent architecture if a simple cascade solves the problem.

The future of AI applications will not be defined only by who has the largest model. It will be defined by who can combine models, tools, retrieval, and human judgment into reliable systems.

Build your first small-model-first cascade, measure every routing decision, and turn model selection into an engineering discipline. Try the examples in this lesson with your own non-sensitive request logs, then continue improving your routing thresholds as you collect real usage data.

Start with a cost-optimized cascade, a quality evaluation set, and a clear escalation policy today.