Context rot: why a bigger context window will not fix your agent

I keep watching teams treat the context window like a bucket. Bigger bucket, more room, fewer problems. Then they point an agent at a task that runs for twenty minutes, pour every tool result and every past message into the prompt, and watch it slowly lose the plot. This is context rot, and it is the most common reason a demo that looked sharp falls apart once the task gets long. The window never filled up. The model just got worse at using what was already in there.

I have been wiring LLM features and small agents into real products for a while now, and the lesson took me a few painful sessions to actually believe. A million token window does not buy you a smarter agent. It mostly buys you more room to confuse one.

What context rot actually is

Context rot is the drop in answer quality as the prompt grows, and it starts well before you hit the token limit. You can be at thirty percent of the window and already see it. The model starts ignoring an instruction it followed five steps ago. It answers based on a stale tool result instead of the fresh one. It contradicts a decision it made earlier in the same run.

The cause is not mysterious. Attention has to spread across everything you hand the model. The more you stuff in, the thinner each token gets weighted, and the easier it is for the important sentence to get drowned out by three hundred lines of JSON you dumped in from an API call. People have measured this across the major models, and the curve bends down long before the advertised limit. The big number on the box is a ceiling, not a promise that quality holds all the way up to it.

Why more tokens stops helping

Once you have seen a long run go sideways, the pattern is always one of these.

• Old information outweighs new. The agent fetched a file, the file changed, it fetched it again, and now both versions sit in the prompt. The model has no clock. It cannot tell which one is current, so it sometimes picks the dead one.
• Tool output buries the goal. A single search or query can return thousands of tokens of noise. The one instruction that matters is now sitting between two walls of logs, and the model reads it like a footnote.
• The agent argues with its past self. Early in the run it decided to use approach A. Halfway through it reconsiders, but the original reasoning is still in context pulling it back, so it flip flops and wastes turns.

None of this gets better by adding more window. You are handing the model a longer book and asking it to remember page two. What it needs is a shorter, cleaner book.

Treat context as a budget you spend, not a bucket you fill

The mental shift that fixed most of my agent reliability was to stop asking how much I can fit and start asking what earns its place. Every token in the prompt is competing for attention with every other token. So I spend the budget on purpose.

• Summarize and compact. When a sub task finishes, I replace the full transcript of it with a short summary of what happened and what the agent learned. The raw back and forth is gone. The conclusion stays.
• Retrieve instead of carry. I do not keep the whole knowledge base in the prompt. I pull the few chunks that matter for the current step, which is the same discipline I wrote about in making RAG actually work. Relevance beats volume every time.
• Trim dead tool output. Once the agent has extracted the three fields it needed from a giant API response, the rest of that response is just noise. I drop it and keep the extract.

There is a cost angle here too. Every stale token you carry is a token you pay for on the next call, on every call, for the rest of the run. Context discipline and a sane bill turn out to be the same project, which is part of why I keep LLM cost under control by watching what goes into the prompt rather than just what comes out.

How I structure a long-running agent now

The shape I trust looks less like one giant conversation and more like a working set with a memory behind it. The prompt holds the current goal, the recent few steps, and whatever I just retrieved for this exact task. Everything else lives outside the window in a store the agent can query when it needs to, and gets summarized back in as a compact note rather than a full replay. That separation between a small live context and a larger external memory is the same idea I dug into in agent architecture basics, and it matters more the longer the agent runs.

The point is that the model should never have to read its whole history to make the next move. It reads a tight, current slice, and trusts the memory layer to surface anything older that turns out to be relevant.

The cheap test most teams skip

Before you blame the model or reach for a bigger window, measure quality against context length on your own task. Run the agent, log how full the context is at each step, and note where the answers start drifting. You will usually find a point where it reliably falls apart, and it is almost always lower than you expected. That number is your real working budget. Build the agent to stay under it.

Most agent failures I get called in to fix are not a weak model. They are a model drowning in its own history. Keep the context short and current, lean on retrieval and summaries for everything else, and the same model that looked unreliable starts behaving like the one in the demo. If you want more on this, I keep related notes in my AI writing, and the rest is on the homepage.