What breaks when you cancel a streaming LLM response

The stop button on a chat UI is a small lie. A user clicks it, the tokens stop scrolling, and everyone assumes the work stopped too. Most of the time it did not. When you cancel a streaming LLM response the way a lot of apps do it, all you really cancelled was your own side of the pipe. The model keeps generating on a GPU somewhere, the provider keeps counting tokens, and your bill keeps climbing for output nobody is going to read. I found this out the boring way, staring at usage numbers that did not line up with what people were actually doing on screen.

Starting a stream is the easy half. You open a connection, you flush tokens as they arrive, the text types itself out, everyone is happy. Stopping one cleanly is where the sharp edges hide, and they hide well because in a demo you never hit them. Nobody cancels in a demo. Real users cancel constantly, and they do it in the messiest possible way, at the worst possible moment.

The stop button only closes your end

Here is the shape of the bug. The browser holds an open connection to your server, usually server-sent events over a plain fetch. When the user hits stop, you abort that fetch, the connection closes, and the UI goes quiet. But your server was in the middle of its own request to the model provider, and closing the browser connection does nothing to that. Unless you explicitly forward the cancellation upstream, the provider request runs to completion and you pay for every token it produces after the user walked away.

The fix is to thread an abort signal all the way through. The client aborts, your server notices the client is gone, and your server aborts its request to the provider using whatever cancellation the SDK gives you. Most provider SDKs take a signal or an equivalent. If you are not passing one, you do not have cancellation. You have a stop button that only stops the drawing.

You find out late that the client left

The annoying part is that detecting a gone client is not instant and it is not clean. TCP will happily sit in a half-open state where your side thinks the connection is fine. You often only discover the client is gone when you try to write the next chunk and the write fails. Put a buffering reverse proxy in the path and that discovery slips even later, because the proxy is still holding a connection to you long after the browser dropped.

So accept that there is always a small window where you keep generating after the user is gone. You cannot make it zero. You can make it small by checking the abort state between chunks instead of only when a write fails, and by turning off proxy buffering on the streaming route. A few wasted tokens per cancel is fine. A whole wasted completion per cancel is a line item.

A dropped connection is not the same as a stop

This one caught me because it looks solved when it is not. On the wire, a user deliberately hitting stop and a user losing their wifi look almost identical. The connection closes either way. If you treat every closed connection as an intentional cancel, then somebody on a flaky train connection loses a half-finished answer they actually wanted, and there is no way to get it back. This is the same trap I wrote about with a real-time UI after a websocket reconnect, where a drop and a deliberate close need different handling.

What works for me is making intent explicit. A real stop sends a small signal before it closes, a request to the server that says cancel this generation. A bare transport drop gets treated as maybe temporary, so the server can hold the partial result for a short window instead of throwing it away. The transport telling you the connection ended is not the same as the user telling you to quit.

The half-written message has to go somewhere

When a stream gets cancelled you are left holding a partial message. The model wrote three sentences and then you cut it off. You have to decide what that partial means. Do you save it, discard it, or store it with a flag that says this was interrupted. My default is to persist it and mark it incomplete. The user watched those words appear, so they expect them to still be there when they scroll back, and quietly deleting text people just read feels broken. Marking it incomplete also lets you do the obvious next thing, which is offer to continue from where it stopped.

The dangerous cancel is the one mid-tool-call

Cancelling while the model is just emitting text is cheap. Cancelling while it is in the middle of calling a tool that changes real state is where you can actually hurt something. If the abort lands after the tool started its work but before it finished, you can end up with a half-applied side effect and no clean record of it. This is exactly why I lean so hard on idempotent tool calls. If running the action twice lands in the same place as running it once, then a cancel in the danger window is survivable. You can retry or roll forward without being scared of what the interrupted call left behind.

Watch the race after the user restarts

Here is the last one, and it is sneaky. A user hits stop and then immediately types a new message and sends it. Your cleanup for the cancelled stream is still running in the background, and if it writes to the same conversation, it can stomp on the new response or land out of order. The cure is a generation id on every stream. Every write carries the id of the stream that produced it, and the server drops any write whose id is no longer the active one. Late writes from a dead stream get ignored on arrival instead of corrupting the live one. It is the same discipline I use choosing SSE or websockets for real-time features, where you assume messages can show up late and out of order and you design so that late arrivals cannot do damage.

What I wire up before I ship a stream

The list is short and I do not skip it anymore. The client holds an AbortController and a real stop sends an explicit cancel before it closes. The server forwards that cancel to the provider so generation and billing actually stop. Every stream carries a generation id and stale writes get dropped. Partial output is persisted and marked incomplete, not thrown away. Any tool the model can call mid-stream is idempotent so a cancel in the wrong spot is safe. None of this is exotic. It is just the difference between a stop button that stops the work and one that only stops the animation. If you want more of how I think about this kind of plumbing, it is in my AI notes, and the rest of the work is on the homepage.