Durable execution: how I make an AI agent survive a crash mid-run

The first time an agent of mine died halfway through a job, it had already done the expensive part. It had pulled the data, called two paid APIs, written half a plan, and then the process got restarted underneath it. When it came back up it had no idea any of that had happened. So it started over from the top, paid for the same two API calls a second time, and produced a slightly different plan than the one it had abandoned. That is the problem durable execution for AI agents is meant to solve, and I wish I had taken it seriously a run earlier than I did.

Durable execution is a plain idea dressed in a fancy name. You save the agent's progress to real storage after every step that matters, so a crash resumes from the last saved point instead of from zero. Nothing about the model changes. The change is entirely in the plumbing around it, which is where I think most of the real work in agents lives anyway.

Why an agent is worse at this than a normal program

A regular backend job that dies can usually just be run again from the start with no harm done, especially if you did the boring reliability work up front. An agent is different in two annoying ways. Its steps cost money and take seconds each, so redoing them from scratch is slow and expensive in a way a quick database read never is. And its steps are not deterministic. Ask the model the same question after a restart and you might get a different answer, a different tool choice, a different branch. So a naive restart does not just repeat work. It can quietly produce a run that never would have existed if the crash had not happened.

That second part is the one people miss. You are not only trying to avoid paying twice. You are trying to make sure the agent that comes back is continuing the same task, not starting a new one that happens to share a name.

Checkpoint the boundaries, not every token

The instinct is to save everything all the time, and that is a trap. You do not need to persist every thinking step. You need to persist at the boundaries where the agent touched the outside world or made a decision you cannot cheaply recompute. A tool call went out and came back. A step finished and produced a result the next step depends on. Those are your save points. Everything between them is cheap to redo, so let it be redone.

In practice I keep a small record per run: an ordered log of completed steps, each with its inputs, its output, and a status. When a step is about to run, I check the log first. If it is already there with a result, I skip the call and hand back the stored answer. If it is not, I run it and append the result before moving on. On a crash, replaying the log rebuilds the agent's state up to the exact step it died on, and it carries on from there. This is the same journal-and-replay pattern that frameworks like LangGraph and Temporal have made their whole pitch, and it is the same shape as a write-ahead log in a database. None of it is new. It is just newly pointed at a language model.

The order that saves you: do the work, then record it, carefully

Here is the part that bites. If you record a step as done before the work actually commits, a crash in that gap leaves you thinking something happened when it did not. If you record it after, a crash in that gap leaves you thinking nothing happened when it did. There is no ordering that makes the window disappear. So you have to make replaying a step safe even when you are not sure it finished.

That is why durable execution does not stand on its own. It leans hard on idempotent tool calls, where running the same action twice lands in the same place as running it once. With that guarantee, the replay does not have to be perfect. It just has to be safe. This is exactly the reasoning I used building indexers, where handling chain reorgs means writing state that survives being rolled back and reapplied without corrupting anything. Agents restarting mid-run and blockchains rewriting recent history are, underneath, the same problem wearing different clothes.

Where I actually put the state

For anything I want to trust, the step log goes in Postgres, not in memory and not in a file on the box the agent happens to be running on. The whole point is to survive the process, so the state has to live somewhere the process does not. This is the same reflex I lean on with background jobs and queues: the moment work matters, its record belongs in durable storage with a clear status, not in a variable that vanishes when the worker dies. A row with a run id, a step index, inputs, output, and a status column is enough to resume almost anything, and it doubles as the audit trail you will want the first time someone asks what the agent actually did.

What I wire up before I trust a long run

The checklist is short. Every run gets an id and an ordered step log in Postgres. Every step checks the log before it acts and appends its result after. Every tool that changes state is idempotent so a replay cannot double-apply it. And resume is a first-class path I test on purpose, by killing the process mid-run and watching it come back, not something I hope works the one time production needs it.

The unglamorous truth is that the agents people trust with long, real tasks are not the ones with the cleverest prompts. They are the ones that treat a crash as a normal event and pick up where they left off without redoing the parts that cost money. If you want to see how the rest of this fits together, I keep writing about it in my AI notes, and the rest of the work is on the homepage.