Your LLM Stream Disconnected. Did the Answer Finish?
Modern AI applications are increasingly using streaming responses to make interactions feel faster and more natural. Instead of waiting for an entire response, users can see the generated content appearing gradually on their screens.
Technically, this looks simple. An application sends a request to an LLM, receives generated tokens, and forwards them to the frontend using Server-Sent Events (SSE).
But what happens when the user's internet connection drops halfway through the response?
Does the model stop generating? Is the response lost? Can the user reconnect and continue receiving the remaining output without starting the entire request again?
These questions become important when building AI applications that are expected to work reliably in production.
The problem is that many applications treat an LLM generation and its HTTP connection as the same thing. In reality, they are two separate operations and should be managed independently.
Let's understand why this matters and how to design a more reliable streaming architecture.
1. Why Streaming Is More Fragile Than It Looks
Consider an AI assistant built using Python, FastAPI, and an LLM provider.
A user submits a question, and the backend starts generating a response. The application streams the generated content to the browser using SSE.
Everything works perfectly until the network connection suddenly disappears.
Several things may happen:
- The browser loses its connection to the backend.
- The SSE connection closes unexpectedly.
- The model provider may still be generating the response.
- The frontend has received only part of the generated content.
- The user refreshes the page and submits the same question again.
Now the application might generate the same answer twice, consume additional API credits, or lose the original response completely.
The important distinction is that an HTTP connection is temporary. An AI generation can be a much longer-running operation.
A disconnected browser does not necessarily mean that the underlying AI task should stop.
This is the first design consideration when building production-ready AI applications.
2. Separate the Generation from the Connection
A better approach is to manage the AI generation independently of the frontend connection.
Instead of generating the entire response inside the SSE endpoint, create a separate generation process.
For example, when a user submits a request:
- The backend creates a unique generation ID.
- It stores a generation record in the database.
- A background worker starts processing the request.
- Generated events are stored with sequence numbers.
- The frontend connects to an SSE endpoint to receive those events.
The generation can now continue even if the browser disconnects.
A generation record might contain:
| Field |
Purpose |
| run_id |
Unique generation identifier |
| user_id |
Owner of the generation |
| status |
Current generation status |
| created_at |
Request creation time |
| completed_at |
Generation completion time |
| last_event_sequence |
Latest persisted event number |
The generation status can move through different states:
- Queued
- Running
- Completed
- Failed
- Cancelled
This gives the application a clear understanding of what is happening independently of whether a user is currently connected.
It also makes it easier to implement generation history, retry mechanisms, monitoring, and cancellation.
3. Persist Events Instead of Keeping Everything in Memory
One common implementation mistake is storing generated chunks only in an in-memory variable.
This might work during development, but it becomes unreliable when the application runs across multiple workers or instances.
Imagine that Worker A generates the response, but the user's reconnect request reaches Worker B.
If the generated content exists only inside Worker A's memory, Worker B cannot retrieve it.
A better solution is to persist generated events.
For example, an event table could look like this:
| run_id |
sequence |
event_type |
payload |
| run_101 |
1 |
token |
Hello |
| run_101 |
2 |
token |
there |
| run_101 |
3 |
token |
! |
| run_101 |
4 |
completed |
Final status |
Each event receives a unique sequence number within its generation.
PostgreSQL can be used for durable event storage, while Redis can help with notifications and temporary coordination between workers.
The important part is that the generated event should be persisted before it is made available for replay.
This creates a reliable history of what the application has generated.
It also allows the frontend to request events after a particular sequence number instead of depending on the original connection.
4. Reconnect Using SSE and Last-Event-ID
Server-Sent Events already provide a useful mechanism for reconnecting to an event stream.
Every SSE event can contain an ID:
id: 1
event: token
data: Hello
id: 2
event: token
data: there
id: 3
event: completed
data: {"status": "completed"}
When an SSE connection reconnects automatically, the browser can send the Last-Event-ID header containing the last event ID it received.
The backend can use that value to retrieve events after the previous position.
For example:
GET /runs/run_101/events
Last-Event-ID: 2
The backend should now return events beginning after sequence number 2.
Here is a simplified FastAPI implementation:
import asyncio
import json
from fastapi import FastAPI, Header
from fastapi.responses import StreamingResponse
app = FastAPI()
@app.get("/runs/{run_id}/events")
async def stream_events(
run_id: str,
last_event_id: str | None = Header(
default=None,
alias="Last-Event-ID"
)
):
cursor = int(last_event_id or 0)
async def event_generator():
nonlocal cursor
while True:
events = await repository.get_events_after(
run_id=run_id,
sequence=cursor,
limit=100
)
for event in events:
payload = json.dumps(event.payload)
yield (
f"id: {event.sequence}\n"
f"event: {event.event_type}\n"
f"data: {payload}\n\n"
)
cursor = event.sequence
run = await repository.get_run(run_id)
if (
run.status in {
"completed",
"failed",
"cancelled"
}
and cursor >= run.last_event_sequence
):
break
await asyncio.sleep(0.5)
return StreamingResponse(
event_generator(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no"
}
)
Here, repository represents an asynchronous database layer responsible for retrieving generation records and persisted events.
This example demonstrates the central idea: the SSE endpoint retrieves existing events instead of owning the actual LLM generation process.
For production, the polling mechanism can be replaced or supplemented with Redis Pub/Sub, database notifications, or another event notification mechanism.
5. Handle Duplicate Events and Cancellation Properly
Reconnecting is only one part of the problem.
A production application must also handle situations where events are delivered more than once.
For example, the browser might receive event number 15, but the connection drops before the frontend finishes processing it.
After reconnecting, the backend might send event number 15 again.
This is why clients should track the last successfully processed sequence number and ignore events they have already applied.
Another important consideration is cancellation.
A browser disconnect should not automatically be treated as a cancellation request.
There is a difference between:
- The user temporarily losing internet connectivity.
- The user closing the browser.
- The user explicitly clicking a Cancel button.
These situations should not necessarily have the same result.
If the user explicitly cancels a generation, the backend can signal the worker to stop processing.
However, if the connection simply disappears, the application can allow the generation to continue until completion or until a configured timeout is reached.
There should also be a clear policy for worker failures, expired event history, retry limits, and abandoned generations.
Without these controls, a background generation system can become expensive and difficult to maintain.
6. Test Failure Scenarios Before Production
A streaming implementation should not be considered reliable just because it works when the internet connection is stable.
Test the situations that are likely to happen in real applications.
Some useful test cases include:
- Disconnect the browser halfway through generation and reconnect.
- Reconnect after the generation has already completed.
- Restart a background worker while it is processing a request.
- Send multiple simultaneous connections for the same generation.
- Submit duplicate requests and verify that idempotency rules work.
- Attempt to retrieve events after the event history has expired.
Also consider infrastructure behavior.
Reverse proxies may buffer responses or terminate long-running connections. Application servers may impose timeouts. Load balancers may distribute reconnect requests to different instances.
These details can affect streaming reliability even when the application code itself is correct.
A reliable implementation must consider the entire path between the model provider and the user's browser.
Final Thoughts
Streaming makes AI applications feel responsive, but responsiveness and reliability are two different engineering problems.
An application that can stream tokens successfully is not automatically capable of recovering from interrupted connections.
The important architectural decision is to separate the lifecycle of an AI generation from the lifecycle of an HTTP connection.
Persisting events, assigning sequence numbers, supporting replay, and managing background workers make it possible to build a much more reliable system.
One thing worth remembering is that resuming event delivery is not the same as resuming model inference. SSE can help replay previously generated events, but it does not automatically restart an interrupted model-provider stream from its previous token.
For production applications, this distinction matters.
A good AI application should not depend on a user's internet connection remaining active throughout the entire generation process. It should know what was requested, what has already been generated, what remains available, and whether the operation has actually completed.
That is the difference between implementing an LLM streaming feature and engineering a reliable AI application.
Suggested tags: Python, FastAPI, Artificial Intelligence, Generative AI, Server-Sent Events, Backend Development, Distributed Systems, Software Architecture
Join the conversation! Your thoughts help the community grow.