A Few Years of Streaming AI
Muhammad Safee Ullah · · 9 min read
From hand-written SSE parsers to streams that outlive the connection carrying them.
Bad internet is not an edge case in Pakistan. For a lot of the people we build for, it is simply what the web feels like.
Some spend the day moving in and out of coverage. Others sit on a line that drops, crawls, and comes back whenever it feels like it. Clean office Wi-Fi is not the environment we design for.
A five-minute research job can involve several model calls and several tool calls. Four minutes in, the stream drops. The user presses Retry and the whole thing starts from zero.
Same commute. Same dead zone. Same bad line. Nothing about pressing that button improves the conditions that killed the first attempt.
Waiting for the first token.
Waiting for the first token.
No work has started.
Work units are simulated, one per token. Restoring the signal counts as pressing Retry on the left.
That bothered me. The people with the worst connections had to sit through the most attempts, and every fresh generation meant paying to repeat work that might already have finished.
I have been building streaming AI chat since the days of hand-written client parsers. Getting an answer onto the screen has become much easier. Making sure someone actually gets to keep it took considerably more work.
When streaming itself was the achievement
SSE was old technology. Using it for AI chat still involved a lot of plumbing: parsing events across arbitrary network chunks, rebuilding messages, and deciding what to do with a half-finished answer.
Things a small hook handles today used to be code we had to write and maintain ourselves.
The Vercel AI SDK was young. LangChain was comfortable in Python and rough in TypeScript. I remember fighting type propagation through chains and working around abstractions that were supposed to be saving us time.
Long-running work had its own problems. Traditional serverless functions had execution limits, and holding an invocation open while a model or tool worked could mean paying for a function to mostly wait.
A dedicated service was one way out, with more infrastructure to own. Compute options like Fluid Compute would improve that tradeoff later. Neither arrangement could keep a phone connected through a dead zone.
The two-faced stream
A proxy can time out or a browser can go to sleep while the original job is still running. Asking the model to generate it again can leave you paying for two attempts at once.
At one point I was explaining a "two-faced stream" to my teammates: one face doing the work, another face showing it to the user.
I can still picture the empty faces looking back at me.
It was more complicated than piping model output down an SSE response. The work needed its own identity and somewhere to keep its progress. The browser connection could then disappear without taking the job with it.
But reconnecting to the right job is only part of the problem. The browser also needs to know where to pick up the answer.
If it rendered the first four events before disconnecting, replaying all ten duplicates what is already on screen. Sending only the newest event loses the middle.
A cursor gives the browser a place to resume. Each complete event gets an ordered id, and the client asks for whatever came after the last one it processed.
The run log and browser are empty.
Grey cells arrived live. White cells arrived through replay. The underlined cell is the cursor: the id of the last complete event the browser processed.
The funny part is that SSE already has a mechanism for this. Events can carry an id, and a reconnecting EventSource sends the last one back in a Last-Event-ID header:
id: 4
data: {"type":"text-delta","text":"checks the evidence,"}
id: 5
data: {"type":"tool-call","name":"search"}
Native EventSource uses GET. For chat APIs that need a POST body, using a fetch stream means carrying the cursor yourself. Either way, the header is only useful if the server still has the events to replay.
Before the first event
There is another awkward case: the server accepts the request, starts the job, and the acknowledgement never reaches the browser. Nothing has appeared on screen. The user has every reason to think nothing happened.
- No traffic yet.
One person, one request. Nothing is running yet.
The key is any stable id the client generates once per user intent, before the first attempt, and reuses on every retry.
This is where the request needs an identity too. The client generates an id once, before the first attempt, and reuses it on retries. The server can recognize the same request and return the existing run instead of starting another.
An event cursor cannot settle that uncertainty on its own. It tells you where you were in an answer, assuming you know which answer you were waiting for.
Deleting the custom plumbing
For a while I was maintaining custom wrappers around the Vercel AI SDK, with patches to make recovery work. As LangGraph matured, I could start deleting that code.
Today, LangGraph's Agent Server and SDK handle much of that execution and streaming plumbing, including persistence and reconnecting to an active run.
I was very happy to delete it. Whenever an issue came up, that layer was another thing I had to load back into my head. Getting rid of it meant less maintenance and less context to carry around.
For someone moving in and out of coverage, a dropped connection is ordinary. It should not make four minutes of useful work disappear.
They should just get their answer.
Historical references: Introducing the Vercel AI SDK · Introducing Fluid Compute · Server-Sent Events in the HTML spec