Streaming responses: tokens, server-sent events, and perceived latency

Imagine two AI assistants answering the same question. Both require eight seconds to generate a complete response.

The first displays a loading spinner for eight seconds, then reveals the entire answer. The second begins displaying useful text after 400 milliseconds and continues writing while generation is still underway.

The underlying computation might be nearly identical, but the user experience is dramatically different.

This distinction matters when building AI assistants, developer tools, analytics dashboards, search interfaces, and other interactive applications. Users don’t judge performance exclusively by how long a request takes to finish. They also notice how quickly useful information appears and whether the interface continues making progress.

Streaming responses improve perceived responsiveness by delivering content incrementally rather than waiting for the entire result.

But streaming introduces engineering challenges. HTTP buffering can prevent incremental delivery, browser clients need explicit completion signals, and measuring the first network byte is not the same as measuring the first usable piece of content.

In this tutorial, we’ll explore how token generation and Server-Sent Events (SSE) work, then build a runnable FastAPI application that compares a streaming endpoint with a conventional blocking response.

We’ll use modern Python dependencies, implement an asynchronous measurement client with aiohttp, validate successful and unsuccessful stream completion, and benchmark first-content latency using repeated measurements and percentile statistics.

Finally, we’ll examine a surprising 2026 usability study showing why the fastest response isn’t necessarily perceived as the most thoughtful.

1. Understanding latency: completion time is only part of the story

A traditional HTTP endpoint often follows this sequence:

1. The client sends a request.

2. The server processes the request.

3. The server constructs the complete response.

4. The client receives the response.

5. The application renders the result.

When the server waits until all work is finished before sending a response body, the user receives no answer content during processing.

Streaming changes that sequence.

Instead of accumulating the complete output, the server sends partial results as they become available. The client can process and display those results before the operation finishes.

This creates several distinct latency measurements.

Time to first byte (TTFB)

Time to first byte measures the elapsed time between initiating a request and receiving the first byte of the HTTP response.

It captures relevant network and server overhead, including connection establishment when a new connection is required.

However, TTFB doesn’t necessarily indicate when the user receives meaningful information.

For example, a streaming endpoint might deliver HTTP headers quickly but take several additional seconds to produce its first content event.

A proxy could also send an initial response promptly while buffering subsequent application data.

TTFB is useful, but insufficient on its own.

Time to first token (TTFT)

Time to first token measures the elapsed time between initiating a generation request and receiving the first generated token.

In language-model applications, TTFT captures the initial waiting period before generation becomes visible to the consumer.

However, terminology matters.

There are at least three potentially different timestamps:

• Model TTFT: When the inference system produces its first token.

• Client first-content latency: When the client receives the first meaningful generated fragment.

• First-visible-content latency: When the interface actually displays that fragment.

These timestamps may differ because of network transmission, buffering, serialization, and browser rendering.

Our benchmark will measure client first-content latency, not internal model TTFT or browser paint timing.

Time per output token (TPOT)

Time per output token describes the average time spent generating subsequent tokens after the first token.

An approximate calculation is:

TPOT = (generation completion time − first-token time) / (number of output tokens − 1)

This assumes at least two generated tokens and consistent timing boundaries.

For example, an assistant that starts answering after 200 milliseconds but then generates only one token per second may still feel frustratingly slow.

TTFT and output throughput should therefore be considered together.

Time to last byte (TTLB)

Time to last byte measures the elapsed time until the complete HTTP response has been received.

For a streaming protocol, it’s also useful to measure application-level completion: the arrival of an explicit event indicating that the operation finished successfully.

These events don’t necessarily have the same timestamp. The server might send a completion event and keep the connection open briefly before closing it.

A practical comparison

Consider an illustrative scenario in which both endpoints perform approximately eight seconds of generation work.

These numbers are hypothetical, not measurements from the application we’re about to build.

Notice that streaming isn’t necessarily faster at completing the operation. It may introduce a little additional overhead.

Its main benefit is making useful output available earlier.

The practical optimization goal is not always to finish all computation sooner. It’s to reduce the delay before the user can begin doing something useful.

2. How token streaming works

Large language models typically generate output through an autoregressive process.

At a simplified level:

1. The model processes the input prompt and available context.

2. It predicts the next token.

3. The generated token becomes part of the context for subsequent generation.

4. The model predicts another token.

5. Generation continues until a stopping condition is reached.

A token is not necessarily a word.

Depending on the tokenizer, it can represent an entire word, part of a word, punctuation, whitespace, or another unit of text.

A model-serving system can expose generated output incrementally rather than accumulating the complete answer.

The application server then relays those fragments to the client.

For example, consider this illustrative delivery timeline:

The frontend can progressively append the incoming fragments to the response being displayed.

Importantly, an API stream chunk doesn’t necessarily correspond to exactly one model token.

Model providers and networking systems may batch several tokens into one chunk, divide encoded content across multiple transport reads, or deliver data at irregular intervals.

A production system should therefore avoid treating every network chunk as a model token.

This distinction becomes important when calculating throughput, interpreting latency metrics, and designing incremental rendering.

3. Server-Sent Events: an HTTP-native streaming protocol

Server-Sent Events provide a standardized mechanism for delivering a sequence of events from a server to a client over HTTP.

SSE is particularly useful when the primary communication direction is from server to client.

Common applications include:

• AI-generated text and explanations.

• Progress updates for long-running operations.

• Live log and telemetry streams.

• Analytics dashboards.

• Background job notifications.

• Search and recommendation results that arrive incrementally.

SSE uses the response media type text/event-stream.

Each event is a sequence of text fields separated by newline characters. An empty line terminates the event.

A typical stream looks like this:

event: token
data: {"text": "Hello"}

event: token
data: {"text": " world"}

event: done
data: {}

The client receives two content events followed by an explicit completion event.

Understanding SSE fields

The standard event-stream format defines several useful fields.

The event field lets the application distinguish content from status, completion, and error notifications.

For example, a useful application-level protocol might define:

• token: A fragment of generated output.

• done: Successful generation completion.

• error: A failure after streaming has started.

• progress: Information about an ongoing operation.

These are application-defined names, not mandatory SSE event types.

A well-designed stream should explicitly indicate successful completion.

Otherwise, a network connection closing unexpectedly could be confused with a successfully finished operation.

The authoritative specification is the WHATWG HTML Living Standard’s server-sent events section.

SSE versus WebSockets

SSE and WebSockets both support incremental delivery, but they have different communication models.

SSE doesn’t prevent a client from making additional HTTP requests. It means the individual SSE connection delivers messages from the server to the client.

For many AI chat applications, that’s sufficient: submit a request, receive generated content incrementally, and close the stream when complete.

For applications involving frequent bidirectional messages or binary media, WebSockets may be more appropriate.

4. Set up a modern Python streaming application

We’ll now build a small application with two HTTP endpoints:

• /stream generates text incrementally and delivers it as SSE events.

• /blocking performs the same generation work but returns a complete JSON response only after generation finishes.

We’ll also add a health-check endpoint and two fault-injection options to test error handling.

Instead of requiring an external AI provider, we’ll simulate generation using asynchronous delays.

This gives us a repeatable workload without API keys, rate limits, unpredictable model output, or inference costs.

Step 1: Choose current dependencies

For this tutorial, use Python 3.14.8, released on September 30, 2026, or another Python 3.14 maintenance release.

Python 3.10 reached end of life on October 1, 2026, so it should not be the recommended runtime for a new production project.

Python 3.14 is actively supported, and the dependencies below advertise compatibility with it. If you choose an even newer interpreter, verify the dependency combination before deployment.

Our dependency baseline, verified against PyPI as of October 16, 2026, is:

These are concrete version pins, rather than minimum constraints that permit substantially older releases.

You can verify their published versions and package requirements on FastAPI’s PyPI page, Uvicorn’s PyPI page, and aiohttp’s PyPI page.

The example targets this environment. Its measured timing remains machine-dependent, and a deployment should additionally lock transitive dependencies after compatibility testing.

Step 2: Create the project

On Linux or macOS, create a directory and virtual environment using Python 3.14:

mkdir sse-latency-lab
cd sse-latency-lab

python3.14 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip

Install the pinned dependencies:

python -m pip install \
  "fastapi==0.143.2" \
  "uvicorn==0.54.0" \
  "aiohttp==3.14.5"

Verify the environment:

python --version
python -m pip show fastapi uvicorn aiohttp

The project will contain two application files:

sse-latency-lab/
├── app.py
├── client.py
└── .venv/

We’ll use the following standard-library modules:

• asyncio for cooperative asynchronous waits.

• time.perf_counter for high-resolution elapsed-time measurements.

• statistics for median calculations.

• math for percentile indexing.

• json for decoding SSE event payloads.

No external model, database, or sample dataset is required.

Step 3: Use FastAPI’s native SSE interface

FastAPI introduced built-in SSE support in version 0.135.0.

Modern releases provide two dedicated types through the fastapi.sse module:

• EventSourceResponse, which sends correctly formatted event streams.

• ServerSentEvent, which represents an event with fields such as event, data, id, and retry.

Using these types is preferable to manually assembling event frames for a new application targeting this version.

The native interface also provides:

• The correct text/event-stream response type.

• JSON serialization for event data.

• Keep-alive comments during periods without events.

• Useful response headers for caching and compatible reverse proxies.

FastAPI’s official SSE documentation describes these behaviors and the supported interface.

Step 4: Implement the complete server

Create app.py with the following code:

import asyncio
from collections.abc import AsyncIterable, AsyncIterator

from fastapi import FastAPI, Request
from fastapi.sse import EventSourceResponse, ServerSentEvent

app = FastAPI()

MESSAGE = (
    "Streaming lets readers see useful text before the server has completed "
    "the entire response. The total work can remain the same, but the initial "
    "wait becomes shorter because content arrives incrementally."
)

PARTS = MESSAGE.split(" ")

INITIAL_DELAY = 0.4
BETWEEN_PARTS_DELAY = 0.08


async def generate_parts() -> AsyncIterator[str]:
    """Simulate incrementally generated text (words, not model tokens)."""
    await asyncio.sleep(INITIAL_DELAY)

    for index, word in enumerate(PARTS):
        if index:
            await asyncio.sleep(BETWEEN_PARTS_DELAY)

        yield word + (" " if index < len(PARTS) - 1 else "")


@app.get("/health")
async def health() -> dict[str, bool]:
    return {"ok": True}


@app.get("/stream", response_class=EventSourceResponse)
async def stream(
    request: Request,
    fail_after: int | None = None,
    truncate_after: int | None = None,
) -> AsyncIterable[ServerSentEvent]:
    sent = 0

    async for part in generate_parts():
        if await request.is_disconnected():
            return

        # Fault injection for demonstration only, not production behavior.
        if fail_after is not None and sent == fail_after:
            yield ServerSentEvent(
                event="error",
                data={"message": "Simulated generation failure"},
            )
            return

        if truncate_after is not None and sent == truncate_after:
            return

        yield ServerSentEvent(
            event="token",
            data={"text": part},
        )
        sent += 1

    yield ServerSentEvent(
        event="done",
        data={},
    )


@app.get("/blocking")
async def blocking() -> dict[str, str]:
    parts = [part async for part in generate_parts()]
    return {"text": "".join(parts)}

Let’s examine the implementation.

The shared asynchronous generator

The generate_parts() function simulates an incremental text-generation service.

It waits 400 milliseconds before producing the first fragment.

After that, it waits 80 milliseconds between fragments.

Because both endpoints consume the same generator, they perform the same simulated generation work.

The message is divided into words purely for demonstration. These words should not be confused with actual language-model tokens.

Our fixed message contains 31 fragments.

Ignoring framework, scheduling, and networking overhead, the expected generation schedule is:

First fragment = 400 ms

Last fragment = 400 ms + (31 − 1) × 80 ms = 2,800 ms

This allows us to predict the general pattern without inventing benchmark measurements.

The streaming endpoint

The streaming endpoint returns an asynchronous sequence of ServerSentEvent objects.

FastAPI serializes each event into the SSE wire format.

For example:

event: token
data: {"text": "Streaming "}

event: token
data: {"text": "lets "}

event: token
data: {"text": "readers "}

JSON serialization also handles special characters in the data safely, including newline characters that would otherwise need attention when constructing the event format manually.

When generation finishes, the endpoint yields a done event.

That event is essential for distinguishing successful completion from unexpected termination.

The blocking endpoint

The blocking endpoint consumes the entire generator before returning its result.

Its response is a single JSON document:

{
  "text": "Streaming lets readers see useful text before the server has completed the entire response. The total work can remain the same, but the initial wait becomes shorter because content arrives incrementally."
}

It performs the same work but doesn’t make partial generated content available to the client.

Simulated failures and incomplete streams

The streaming endpoint also accepts two optional query parameters for local fault injection.

The first causes an application-level failure:

curl -N "http://127.0.0.1:8000/stream?fail_after=3"

After three fragments, the endpoint emits:

event: error
data: {"message": "Simulated generation failure"}

The second simulates a stream that ends without a successful completion event:

curl -N "http://127.0.0.1:8000/stream?truncate_after=3"

In this case, the endpoint sends three fragments and then closes normally at the HTTP transport level without sending done.

This deliberately malformed application-level exchange helps us test an important production requirement: HTTP completion must not automatically be interpreted as successful generation completion.

These fault-injection parameters exist only for this tutorial. They should not be exposed as public controls in a production application.

5. Start the server and inspect the events

Open your first terminal and start Uvicorn:

python -m uvicorn app:app --host 127.0.0.1 --port 8000

Keep the server running.

In another terminal, inspect the health endpoint:

curl -i http://127.0.0.1:8000/health

You should receive a successful HTTP response containing a JSON object indicating that the application is running.

Next, inspect the streaming endpoint:

curl -N -i http://127.0.0.1:8000/stream

The -N flag disables curl’s output buffering.

After the HTTP headers, SSE events should arrive incrementally.

The first few events should resemble:

event: token
data: {"text": "Streaming "}

event: token
data: {"text": "lets "}

event: token
data: {"text": "readers "}

The complete stream eventually finishes with a done event.

Now inspect the blocking endpoint:

curl -i http://127.0.0.1:8000/blocking

Unlike the streaming endpoint, the blocking endpoint won’t deliver its usable JSON body until all simulated generation has finished.

This difference is visible even though the generation workloads are equivalent.

However, watching text appear in a terminal isn’t sufficient for reliable performance analysis.

We need to measure the application-level timing.

6. Build an asynchronous SSE measurement client

Our measurement client will compare two metrics for each endpoint:

1. Time until the first meaningful generated content becomes available.

2. Time until the application reports successful completion.

For the streaming endpoint, the first measurement occurs after parsing the first nonempty token event.

Successful completion is recorded only after parsing done.

For the blocking endpoint, the first usable result and complete result become available together, once the JSON response has been received and decoded.

We’ll use aiohttp 3.14.5 for asynchronous HTTP requests.

The library’s streaming API documentation confirms that iterating over a response’s content stream reads lines by default.

That’s useful for our protocol parser because SSE messages are line-oriented, while the underlying network may deliver arbitrary byte chunks.

Step 1: Implement the full measurement client

Create client.py:

import asyncio
import json
import math
from statistics import median
from time import perf_counter

import aiohttp

BASE_URL = "http://127.0.0.1:8000"
RUNS = 12


async def measure_stream(
    session: aiohttp.ClientSession,
    path: str = "/stream",
) -> tuple[str, float, float]:
    started = perf_counter()
    first_content_at: float | None = None
    fragments: list[str] = []
    event_name = "message"
    data_lines: list[str] = []

    async with session.get(
        path,
        headers={"Accept": "text/event-stream"},
    ) as response:
        response.raise_for_status()

        if "text/event-stream" not in response.headers.get(
            "Content-Type", ""
        ):
            raise RuntimeError("Server did not return an SSE stream")

        # aiohttp's response.content iterator reads complete lines.
        async for raw_line in response.content:
            line = raw_line.decode("utf-8").rstrip("\r\n")

            if line == "":
                if data_lines:
                    payload = json.loads("\n".join(data_lines))

                    if event_name == "token":
                        part = payload["text"]

                        if part:
                            if first_content_at is None:
                                first_content_at = perf_counter()

                            fragments.append(part)

                    elif event_name == "error":
                        raise RuntimeError(
                            f"Stream error: "
                            f"{payload.get('message', 'unknown')}"
                        )

                    elif event_name == "done":
                        if first_content_at is None:
                            raise RuntimeError(
                                "Completion without content"
                            )

                        finished = perf_counter()

                        return (
                            "".join(fragments),
                            (first_content_at - started) * 1000,
                            (finished - started) * 1000,
                        )

                event_name = "message"
                data_lines = []
                continue

            if line.startswith(":"):
                continue

            field, sep, value = line.partition(":")

            if not sep:
                continue

            if value.startswith(" "):
                value = value[1:]

            if field == "event":
                event_name = value

            elif field == "data":
                data_lines.append(value)

    raise RuntimeError(
        "SSE connection closed before the done event"
    )


async def measure_blocking(
    session: aiohttp.ClientSession,
) -> tuple[str, float, float]:
    started = perf_counter()

    async with session.get("/blocking") as response:
        response.raise_for_status()
        payload = await response.json()

    finished = perf_counter()
    elapsed_ms = (finished - started) * 1000

    return payload["text"], elapsed_ms, elapsed_ms


def percentile(samples: list[float], rank: float) -> float:
    ordered = sorted(samples)
    index = math.ceil(len(ordered) * rank) - 1
    return ordered[index]


def report(name: str, samples: list[float]) -> None:
    print(
        f"{name:<26} "
        f"p50={median(samples):7.0f} ms  "
        f"p95={percentile(samples, 0.95):7.0f} ms"
    )


async def main() -> None:
    results: dict[str, list[float]] = {
        "Streaming first content": [],
        "Streaming completion": [],
        "Blocking first content": [],
        "Blocking completion": [],
    }

    timeout = aiohttp.ClientTimeout(total=15)

    async with aiohttp.ClientSession(
        base_url=BASE_URL,
        timeout=timeout,
    ) as session:
        async with session.get("/health") as response:
            response.raise_for_status()
            await response.json()

        for path in (
            "/stream?fail_after=3",
            "/stream?truncate_after=3",
        ):
            try:
                await measure_stream(session, path)

            except RuntimeError as exc:
                print(f"Correctly rejected {path}: {exc}")

            else:
                raise AssertionError(
                    f"Fault was not detected: {path}"
                )

        # The first pair is a warm-up.
        # Alternate request order afterward.
        for run in range(RUNS + 1):
            if run % 2 == 0:
                streamed = await measure_stream(session)
                blocked = await measure_blocking(session)

            else:
                blocked = await measure_blocking(session)
                streamed = await measure_stream(session)

            if streamed[0] != blocked[0]:
                raise AssertionError(
                    "Endpoint output differs"
                )

            if run > 0:
                results["Streaming first content"].append(
                    streamed[1]
                )
                results["Streaming completion"].append(
                    streamed[2]
                )
                results["Blocking first content"].append(
                    blocked[1]
                )
                results["Blocking completion"].append(
                    blocked[2]
                )

    print(
        f"\n{RUNS} timed pairs "
        f"(after one warm-up pair):"
    )

    for name, samples in results.items():
        report(name, samples)

    improvement = (
        median(results["Blocking first content"])
        / median(results["Streaming first content"])
    )

    print(
        f"Median first-content improvement: "
        f"{improvement:.1f}x"
    )


if __name__ == "__main__":
    asyncio.run(main())

This client is deliberately more careful than a simple streaming demonstration.

Let’s examine why.

Step 2: Parse complete SSE events, not arbitrary network chunks

HTTP transports deliver bytes, not application-level event objects.

A network read might contain:

• Part of an SSE field.

• Exactly one event.

• Several complete events.

• A comment followed by part of another event.

Our client iterates over complete lines and accumulates data fields until it encounters the blank line marking the end of an SSE event.

It then decodes the JSON payload.

This approach avoids assuming that every transport read contains one complete event.

The implementation handles the named events used by our application, comment lines, and multiline data fields. It isn’t intended as a general-purpose implementation of every feature of the SSE specification.

Step 3: Measure first meaningful content

The measurement begins immediately before making the streaming request.

However, we don’t stop the timer when the server sends response headers.

Instead, we wait until a complete token event has been parsed and a nonempty text fragment is available.

That gives us a practical client-side first-content measurement.

It includes the connection and HTTP request overhead for that request, along with server generation, transmission, and client parsing.

It doesn’t include browser rendering time, since this experiment uses a Python client.

Step 4: Validate the final completion event

This is one of the most important improvements over a simplistic SSE client.

The parser recognizes three outcomes:

• Token received: Append the content and continue.

• Error received: Raise an exception instead of reporting success.

• Done received: Record successful completion and return the measurements.

If the connection closes before done arrives, the client raises an exception.

This prevents a truncated response from being counted as a successfully completed generation.

The benchmark also actively tests both fault-injection mechanisms before collecting normal timing samples.

The first test verifies that an explicit error event is rejected.

The second verifies that an event stream ending without done is rejected.

Note that some real network failures can also raise transport-specific exceptions from aiohttp before the parser reaches end-of-stream. Those failures should likewise be treated as unsuccessful requests, not quietly converted into completion times.

Step 5: Measure the blocking response fairly

The blocking endpoint returns a JSON object after generation has finished.

Our client waits until that JSON has been decoded before recording the time.

This means both endpoints are measured at the point where their results can be used by application code.

The comparison intentionally measures different communication behaviors for equivalent simulated generation work.

It doesn’t attempt to compare the raw performance of different JSON and SSE parsers.

Step 6: Run repeated measurements

With Uvicorn running in the first terminal, run:

python client.py

The script first validates both failure scenarios.

It then performs one warm-up pair and 12 timed request pairs, alternating which endpoint runs first to reduce systematic ordering bias.

The output reports:

• Streaming first-content p50 and p95.

• Streaming completion p50 and p95.

• Blocking first-content p50 and p95.

• Blocking completion p50 and p95.

• Median first-content improvement.

Unlike a single hand-picked timing example, these values are calculated from the requests executed on your own machine.

What results should you expect?

The simulation’s fixed timing schedule gives us a useful baseline:

Expected simulation schedule (excluding runtime overhead)

First generated fragment: approximately 400 ms
Last generated fragment:  approximately 2800 ms

Streaming:
  First useful content arrives near the initial delay.
  Completion follows the entire generation schedule.

Blocking:
  First usable response arrives after generation completes.
  Completion occurs at essentially the same point.

This is a theoretical schedule, not fabricated benchmark output.

Your printed p50 and p95 values will include HTTP, event-loop scheduling, serialization, parsing, and operating-system overhead.

The expected outcome is that streaming dramatically improves first-content latency, while completion times remain similar.

For reference, here is one run of client.py against the server above (Python 3.14.6, FastAPI 0.143.2, Uvicorn 0.54.0, aiohttp 3.14.5, localhost, October 16, 2026):

Correctly rejected /stream?fail_after=3: Stream error: Simulated generation failure
Correctly rejected /stream?truncate_after=3: SSE connection closed before the done event

12 timed pairs (after one warm-up pair):
Streaming first content    p50=    402 ms  p95=    404 ms
Streaming completion       p50=   2836 ms  p95=   2841 ms
Blocking first content     p50=   2824 ms  p95=   2829 ms
Blocking completion        p50=   2824 ms  p95=   2829 ms
Median first-content improvement: 7.0x

The numbers track the theoretical schedule closely: about 2 ms of overhead before the first fragment, and roughly 25-40 ms on top of the 2,800 ms generation time. Streaming completion is a few milliseconds slower than the blocking response because of per-event framing and flushing. That is the price of showing the first words about seven times sooner.

Understanding p50 and p95

The script reports two percentiles.

p50, or the median, represents the middle of the measured latency distribution.

p95 represents a higher-latency observation: approximately 95% of measurements fall at or below that threshold under the selected percentile convention.

Our implementation uses the nearest-rank method for p95.

With only 12 samples, the p95 result is the maximum observed value. It’s useful for demonstrating the calculation, but it is not a stable estimate of real production tail latency.

For serious benchmarking, increase the sample count and perform measurements over multiple periods under realistic load.

Keep the percentile definition consistent across monitoring systems, since different tools may use different interpolation methods.

Reproducibility and benchmark limitations

The reproduction environment is defined by:

• Python 3.14.8.

• FastAPI 0.143.2.

• Uvicorn 0.54.0.

• aiohttp 3.14.5.

• Loopback HTTP requests to 127.0.0.1.

• One local Uvicorn server process.

• A fixed, 31-fragment asynchronous generation schedule.

• One warm-up pair followed by 12 timed pairs.

• Sequential requests with alternating endpoint order.

The hardware, operating system, background system load, and installed transitive dependency versions can still influence timings.

Those details should be recorded when publishing an actual benchmark run.

A loopback test also excludes real internet latency, browser rendering, production proxies, CDN behavior, and model inference.

The results should be interpreted as a controlled demonstration of streaming delivery, not a claim about universal production performance.

For more rigorous testing, benchmark the deployed public endpoint, record the environment, repeat under controlled concurrency, and examine distributions rather than relying on a single run.

7. From Python clients to browser interfaces

The Python client gives us controlled timing measurements, but most interactive applications eventually render streamed content in a browser.

Modern browsers support SSE through the native EventSource API.

For a GET-based stream, the integration is relatively straightforward.

A browser application should:

1. Create an EventSource connection to the streaming endpoint.

2. Register a listener for named token events.

3. Parse each event’s JSON payload.

4. Append the text to an accumulated response.

5. Listen for the done event.

6. Explicitly close the EventSource connection on completion.

The final step is important because EventSource supports automatic reconnection.

If a finite generation stream closes without the frontend calling the connection’s close() method, the browser may attempt to reconnect unnecessarily.

That can lead to duplicate generation, unexpected server load, or confusing UI behavior.

See the MDN EventSource documentation for the browser connection lifecycle.

Render generated text safely

When displaying generated content, avoid inserting untrusted text directly as HTML.

Even if output comes from a model you control, the generated text can contain HTML-like syntax, links, or potentially malicious markup.

Prefer safe text insertion methods when displaying plain text.

If your application supports Markdown, use a Markdown renderer with an appropriate security and sanitization policy.

Streaming Markdown introduces an additional complication: incoming fragments may be syntactically incomplete.

For example:

• A code block might begin in one event and close several events later.

• A Markdown link might arrive in multiple fragments.

• A table row could be incomplete during rendering.

• An inline formatting marker might arrive before its matching closing marker.

A robust interface should separate the accumulated raw text from its rendered presentation.

It can periodically render the current content rather than rebuilding the entire document after every small fragment.

Control rendering frequency

Receiving data frequently doesn’t mean the browser must repaint after every event.

For example, imagine a service delivering 100 small events per second.

If the frontend repeatedly reparses a large Markdown document and updates complex UI components for every event, rendering overhead can become substantial.

The application can buffer incoming fragments briefly and schedule display updates at a controlled frequency.

This creates two separate rates:

• Transport rate: How frequently the application receives data.

• Rendering rate: How frequently the interface updates the visible content.

Decoupling these rates can improve responsiveness, especially for large responses.

The goal is not to display every network fragment individually. It’s to make progress visible smoothly without overloading the browser.

8. Production best practices for reliable streaming

A working localhost demonstration doesn’t guarantee reliable streaming in production.

Reverse proxies, compression, authentication, timeouts, and application cancellation all affect how streaming behaves.

Let’s examine the most important deployment considerations.

Avoid reverse-proxy buffering

Response buffering is one of the most common causes of apparently broken streaming.

A server might generate events correctly, while an intermediary accumulates those events before forwarding them to the user.

The result is an endpoint that technically streams but behaves like a blocking response.

Nginx, for example, supports response buffering through its proxy_buffering configuration directive.

It also supports the X-Accel-Buffering response header, which can disable buffering when the relevant configuration honors that header.

FastAPI’s native SSE interface already sets useful headers, including:

• Cache-Control: no-cache

• X-Accel-Buffering: no

However, this doesn’t guarantee that every proxy, CDN, or gateway will forward events immediately.

Consult the official Nginx proxy documentation for the precise behavior of buffering controls.

When troubleshooting, inspect the complete delivery path:

• ASGI application.

• Application server.

• Reverse proxy.

• API gateway.

• Load balancer.

• CDN.

• Browser or HTTP client.

A streaming endpoint should be tested through the actual public route used by customers.

Avoid blocking the asynchronous event loop

The example uses asynchronous sleep operations because they allow the event loop to handle other work while waiting.

That cooperative behavior is essential for concurrent asynchronous applications.

By contrast, directly performing blocking work inside an asynchronous generator can delay other requests handled by the same event loop.

Typical sources of accidental blocking include:

• Synchronous network calls.

• Long-running CPU-bound calculations.

• Blocking database operations.

• Synchronous file-processing loops.

• Model SDK methods that don’t provide an asynchronous interface.

Use supported asynchronous interfaces for I/O when available.

For unavoidable blocking work, consider threads, worker processes, or an architecture that separates inference and response delivery.

Simply adding an asynchronous function declaration doesn’t make blocking code nonblocking.

Handle compression carefully

Compression can reduce bandwidth, but it can also interfere with incremental delivery when an implementation buffers data before producing compressed output.

An SSE response consisting of many small messages can be particularly sensitive to this behavior.

If the uncompressed endpoint streams smoothly but the compressed deployment delivers events in batches, investigate:

• Response compression middleware.

• Reverse-proxy compression.

• Gateway buffering.

• Flush behavior.

• Intermediate caches.

Don’t assume that yielding an event from application code means the browser immediately receives it.

Measure delivery at the client.

Implement sensible keep-alive behavior

Long-running streams may contain periods when no generated content is available.

Some infrastructure closes connections that appear idle.

Keep-alive comments can help maintain activity on an SSE connection without introducing application content.

FastAPI’s native SSE response provides keep-alive comments when a stream has no events for an interval; the documented default is 15 seconds.

These comments don’t represent generated text.

Our benchmark client correctly ignores lines beginning with the comment marker, so keep-alive traffic doesn’t accidentally count as first content.

Keep-alives are useful, but they don’t replace proper timeout configuration.

The server, reverse proxy, gateway, and client must have compatible expectations for long-running operations.

Handle cancellation and disconnects

Users regularly abandon requests before generation finishes.

They might:

• Navigate away from the page.

• Close a browser tab.

• Cancel an operation.

• Submit a replacement question.

• Lose network connectivity.

A production system should avoid continuing expensive work for requests whose consumers have disappeared.

Our demonstration checks whether the request is disconnected.

Real applications should also propagate cancellation to upstream generation tasks where supported.

For example, canceling an abandoned inference request can reduce wasted compute and improve capacity for other requests.

Resource cleanup should cover upstream HTTP responses, open file handles, database connections, and temporary resources.

Design an explicit error protocol

With an ordinary blocking endpoint, the server can return an HTTP error status before transmitting the final response.

Streaming changes this behavior.

Once response headers have been sent, the server generally cannot replace the original HTTP status with a different one to report a later generation failure.

That means errors occurring midstream require an application-level representation.

For example:

event: error
data: {"message": "Generation could not be completed"}

The client should recognize this as a failure, stop consuming the answer, and present appropriate feedback.

Successful completion should use a distinct event:

event: done
data: {}

A disconnected stream without either event should be considered incomplete.

Avoid exposing raw exception messages, credentials, internal URLs, or implementation details in public error payloads.

Server logs can retain detailed diagnostic information while clients receive a safe error description.

Make reconnection behavior intentional

SSE supports event identifiers and reconnection mechanisms.

However, adding an event ID doesn’t automatically make a generation resumable.

To resume reliably, the server must have a way to identify the previous operation and recover or replay the necessary output.

This can be relatively straightforward for a retained event log.

It can be much more complicated for nondeterministic language-model generation.

Restarting a prompt may produce different text, while replaying already displayed events can duplicate content.

A production application should explicitly decide whether interruptions are handled by:

• Resuming the existing operation.

• Replaying retained events.

• Starting a new generation.

• Reporting an interrupted response and requesting user action.

Don’t implement automatic retries without considering duplication and consistency.

Secure authentication and sensitive inputs

The native browser EventSource interface establishes GET requests and doesn’t provide a general mechanism for attaching arbitrary custom authorization headers.

That can matter for applications using bearer tokens or sensitive prompt bodies.

An alternative is to use the browser Fetch API with a POST request and consume the response as a readable stream.

FastAPI’s SSE response interface supports POST as well as GET.

However, when using Fetch, the application must handle SSE framing itself or use an appropriate parser, since Fetch doesn’t automatically dispatch SSE events like EventSource does.

For authenticated deployments:

• Use HTTPS.

• Authorize every streaming request.

• Avoid putting sensitive prompts or access tokens in URLs.

• Apply appropriate cookie, CORS, and CSRF protections.

• Enforce request limits and reasonable resource budgets.

• Treat generated output as untrusted content.

Streaming changes the transport behavior, not the application’s security responsibilities.

9. Choosing the right event granularity

An effective streaming application doesn’t need to send one network event for every model token.

In fact, that may be inefficient.

Extremely small events introduce overhead from framing, serialization, networking, parsing, and frontend updates.

But waiting for very large batches delays useful output and weakens the responsiveness benefit.

The right balance depends on the application.

Consider several examples.

AI-generated text

A text assistant usually benefits from frequent, relatively small updates.

Users can begin reading the response while the model continues generating.

However, individual model tokens might not align with words, sentences, or Unicode character boundaries.

Applications should carefully handle incremental text decoding and avoid making assumptions about chunk boundaries.

Live analytics

A dashboard receiving thousands of telemetry readings per second probably shouldn’t render every individual reading.

Instead, it might aggregate measurements over a short window and transmit compact updates.

The interface can show meaningful trends without overwhelming the browser or network.

Log streaming

For logs, a complete line is often a useful application-level unit.

Delivering a partial line may make the display harder to interpret without providing much benefit.

Background processing

An import job processing 100,000 retail orders doesn’t need to send an event for every order.

It could report milestones such as:

• Input validation completed.

• 25% of records processed.

• 50% of records processed.

• Output validation completed.

• Import finished.

The appropriate granularity is the smallest meaningful unit of progress, balanced against transport and rendering overhead.

Separate transport and presentation decisions

A useful pattern is to deliver events frequently while rendering them at a controlled rate.

This preserves early content delivery without forcing the frontend to perform expensive updates for every fragment.

For AI interfaces, it can also be useful to distinguish text-generation events from other operations such as tool execution, retrieval, or validation.

Meaningful progress events should describe actual work being performed, rather than simulated or misleading activity.

10. Measure streaming performance in production

The local benchmark demonstrates how streaming changes delivery timing.

Production observability requires a broader set of measurements.

At minimum, track:

For model inference, throughput can be measured in tokens per second when actual token counts are available.

For a general-purpose SSE service, bytes per second or application events per second may be more appropriate.

These metrics shouldn’t be treated as interchangeable.

Compare server timing with client timing

Consider a case where model generation starts quickly, but the frontend receives its first event much later.

That timing gap suggests additional delay somewhere between model generation and client consumption.

Potential causes include:

• Serialization.

• Application-side buffering.

• Reverse-proxy buffering.

• Network congestion.

• Client parsing overhead.

• Browser main-thread activity.

Correlating server and client measurements helps identify which layer deserves optimization.

For example, if the model produces its first token after 300 milliseconds but the browser doesn’t display content until 2,000 milliseconds, optimizing model inference alone is unlikely to resolve the entire problem.

The delivery and rendering path needs attention.

Watch tail latency

Averages can hide frustrating behavior.

Imagine two systems with similar average first-content latency.

The first responds quickly most of the time but occasionally stalls for several seconds.

The second starts slightly later but behaves consistently.

Depending on the workload, the second may provide a more predictable experience.

Track distributions, including:

• p50 for typical latency.

• p95 for slower requests.

• p99 for extreme tail behavior.

Analyze these values by relevant dimensions, such as endpoint, model, region, request size, and device category.

Avoid putting raw prompts or other high-cardinality sensitive inputs into telemetry labels.

Measure actual visual progress

Client first-content latency isn’t the same as first-visible-content latency.

A browser might receive a fragment while its main thread is busy processing another task.

The content won’t become visible until the browser has an opportunity to update the display.

For a sophisticated performance study, record meaningful frontend milestones and use browser performance instrumentation to understand when updates actually become visible.

A useful user-experience experiment can combine timing measurements with:

• Task completion rates.

• User satisfaction.

• Abandonment rates.

• Perceived responsiveness.

• Error recovery success.

• Interface interaction behavior.

A fast stream that regularly fails halfway through generation may be worse than a slightly slower but reliable implementation.

11. Cherry on the cake: a 2026 study found that a 2-second response could seem less thoughtful than a 9-second response

Streaming is often discussed as though lower latency automatically produces a better experience.

But the relationship between response speed and human perception is more complicated.

A study published on April 13, 2026, in the Proceedings of the CHI Conference on Human Factors in Computing Systems investigated how response latency influences people’s interactions with large language models.

The paper, The Impact of Response Latency and Task Type on Human-LLM Interaction and Perception, was authored by Felicia Fang-Yi Tan, Moritz Alexander Messerschmidt, Wen Yin, and Oded Nov.

The researchers analyzed 240 participants across two types of knowledge-work tasks: creation and advice.

Participants experienced one of three controlled time-to-first-token conditions.

After the first token, the experiment delivered output at a fixed rate of 25 tokens per second.

This experimental design isolated initial response delay while holding subsequent streaming speed constant.

The surprising result

Participants in the two-second condition rated responses as less thoughtful than those in the nine-second and 20-second conditions.

Responses in the nine-second condition were also rated significantly more useful than those in the two-second condition.

The researchers observed that some participants interpreted longer waits as evidence that the AI was spending more effort deliberating.

The result is interesting because the underlying system’s response timing was experimentally controlled.

A longer delay could influence users’ interpretations of answer quality without demonstrating that the underlying answer had benefited from additional reasoning.

Participants noticed long waits much more frequently

The study also measured whether participants consciously noticed response delays.

In the advice-task condition:

These figures are specific to the advice-task condition, not pooled percentages for all participants.

Longer waits were much more noticeable, yet participants didn’t necessarily translate every additional second into a worse assessment of response quality.

The study also found limits to the perceived benefits of waiting: very long delays sometimes produced frustration or concerns about reliability.

What this means for streaming design

The study was not a direct comparison between a blocking HTTP endpoint and an SSE implementation.

It manipulated TTFT while controlling the subsequent output rate.

Therefore, it doesn’t establish that blocking responses are better or that applications should deliberately delay useful content.

Instead, it demonstrates that response timing communicates information to users, and that information can influence perceived quality.

For a simple factual question, prompt delivery is usually a sensible design goal.

For complex analytical work, users may also value a clear indication of meaningful ongoing processing.

A good interface can communicate actual stages of work, such as retrieving data, generating content, or validating a result.

It shouldn’t fabricate activity or intentionally waste time to appear more thoughtful.

The lesson is to optimize responsiveness alongside transparency, reliability, and useful feedback.

That distinction matters because a faster technical implementation doesn’t automatically produce a better human experience in every context.

12. When streaming is not the right choice

Streaming introduces additional complexity.

It requires event parsing, completion tracking, cancellation handling, and careful infrastructure configuration.

Not every endpoint benefits from these costs.

A conventional JSON response may be preferable when:

• The operation already finishes quickly.

• The output is small.

• The result must be validated in full before display.

• Partial information would be confusing or unsafe.

• The workflow requires an atomic result.

• The deployment infrastructure doesn’t reliably support incremental delivery.

Consider an API returning a small configuration object.

If the complete response is available almost immediately, splitting it into multiple events is unlikely to improve the user experience.

Similarly, some operations shouldn’t expose partial results before validation.

For example, a financial transaction confirmation or an authoritative configuration change needs a clearly established final state.

A partially delivered result could mislead downstream systems.

Streaming versus progress notifications

Sometimes an operation takes a long time but doesn’t produce useful intermediate content.

A large file conversion might require significant computation before the final artifact is ready.

In that case, progress events can still be valuable even if the application doesn’t stream the final file itself.

For example, an interface could report that conversion started, processing reached certain milestones, and validation completed.

The final downloadable artifact would appear only after successful completion.

This is different from text streaming, where the content itself can often be consumed incrementally.

Streaming versus WebSockets

If the application needs frequent bidirectional communication, SSE may not be the best transport.

Examples include interactive collaborative sessions, real-time multiplayer interaction, or systems exchanging rapidly changing state in both directions.

WebSockets may provide a more natural communication model for those workloads.

The choice should follow the application’s communication requirements rather than a blanket preference for one technology.

13. A practical deployment checklist

Before launching a streaming service, review the following areas.

Protocol correctness

• Responses use text/event-stream.
• Events have valid framing and blank-line termination.
• Content events are distinguishable from status events.
• Successful completion has an explicit signal.
• Errors have an explicit application-level representation.
• Unexpected termination is not reported as success.

Application reliability

• Asynchronous generation doesn’t block the event loop.
• Client disconnects are handled.
• Abandoned upstream work can be canceled where supported.
• Network and application resources are cleaned up.
• Timeouts accommodate legitimate generation times.
• Fault injection tests cover failures and missing completion events.

Infrastructure

• Reverse proxies preserve incremental delivery.
• Compression doesn’t unexpectedly buffer output.
• Keep-alive behavior is compatible with network timeouts.
• Authentication and authorization apply to streaming requests.
• Sensitive inputs are not unnecessarily exposed in URLs or logs.
• The public deployed endpoint has been tested, not just localhost.

Frontend experience

• Generated text is rendered safely.
• Browser clients handle completion and errors.
• Completed EventSource connections are closed.
• Reconnection behavior avoids unintended duplicate generation.
• Rendering frequency is controlled.
• Users receive meaningful feedback during long operations.

Observability

• First-content latency is measured separately from TTFB.
• Completion latency is tracked.
• Measurements include p50, p95, and appropriate tail metrics.
• Error and cancellation rates are monitored.
• Server and frontend metrics can be correlated.
• Benchmark environments and dependency versions are recorded.

This checklist won’t guarantee a particular response time, but it helps prevent the familiar situation where streaming works perfectly in development and behaves like a blocking endpoint after deployment.

14. Key takeaways

Streaming responses are a powerful technique for improving responsiveness when an operation cannot finish instantly.

Server-Sent Events provide a standardized, HTTP-native mechanism for sending incremental updates from a server to a client.

Modern FastAPI releases include a dedicated SSE response interface, making it easier to build correctly framed event streams without manually constructing protocol messages.

Our example demonstrated how two endpoints can perform equivalent generation work while providing very different first-content delivery behavior.

The underlying simulation begins producing text after approximately 400 milliseconds and completes generation after approximately 2.8 seconds.

The streaming endpoint exposes content incrementally. The blocking endpoint waits until all content is available.

The benchmark client makes that distinction measurable through repeated requests, median and p95 latency calculations, and explicit validation of successful completion.

The broader engineering lessons are equally important:

• Streaming is primarily about earlier access to useful information, not necessarily shorter overall computation.

• Network chunks, model tokens, and rendered content are different measurement units.

• Reverse proxies and middleware can defeat incremental delivery through buffering.

• Completion and error events should be part of the application protocol.

• Streaming clients must handle premature termination and cancellation.

• Production performance should be evaluated through actual deployment infrastructure.

• User perception depends on clarity and reliability as well as raw speed.

The 2026 CHI study adds a useful reminder: faster responses aren’t automatically perceived as better in every task context.

A successful streaming experience combines early useful output, consistent progress, reliable completion, and transparent communication about what the system is doing.

15. Put streaming into practice

The next step is to validate streaming performance in your own environment.

Start with the runnable FastAPI server and aiohttp measurement client from this tutorial.

Run the experiment with the pinned dependencies, inspect the first-content and completion measurements, and confirm that both injected failure scenarios are correctly rejected.

Then extend the experiment:

1. Replace the simulated generator with a real streaming text source using its currently documented asynchronous interface.

2. Measure model-side TTFT separately from client first-content latency.

3. Test through the reverse proxy and gateway configuration used in deployment.

4. Add frontend instrumentation for the first visible content.

5. Compare p50 and p95 latency against the existing blocking implementation.

6. Monitor failure rates and abandonment alongside responsiveness.

Don’t stop at seeing text appear incrementally in a terminal. Validate that streaming delivers measurable benefits through the full application stack and remains correct when connections fail.

Your next optimization target isn’t necessarily making every request finish sooner. It’s making the first useful result arrive sooner—and proving that improvement with reliable measurements.