Designing Agent State Checkpoints for Crash Recovery
Persisting all four layers of agent state prevents silent failures when production systems restart.
An agent that loses its state the moment it crashes is a different category of software, one that cannot be handed a task longer than a single uninterrupted run without risking total loss of progress. A research agent that spends a long stretch gathering sources, or a coding agent that has spent the same stretch indexing a repository, faces an identical fate on crash: every bit of that work vanishes, and the agent begins again from zero. That is not a tolerable failure mode for anything described as production software. LLMs carry no memory between API calls, so every piece of continuity a long-running agent appears to have is something a developer built, not something the model provides. The agent's state exists entirely in infrastructure the developer controls, so crash recovery is a first-class architectural concern that has to be designed alongside the agent's core logic from the start. LangChain's 2026 State of Agent Engineering report identifies quality, including hallucinations and output consistency, as the top barrier to production deployment, with context engineering at scale also cited as a major challenge among large enterprises. Neither of those problems can be solved by an agent that cannot even reliably hold onto the context it has already built. The rest of this piece lays out what a checkpoint architecture has to capture, where the obvious implementations of that architecture break, and how to build one that treats serialization, storage placement, and resume logic as a single coherent system.
The four layers of agent state that a checkpoint must capture
Agent state is not one thing with one shape. It spans at least four distinct layers, and a checkpoint that captures only some of them does not produce partial recovery so much as recovery that fails silently on exactly the layers it missed.
The first layer is in-context working memory: everything currently sitting inside the model's context window, including conversation history, the results of tool calls, and the system instructions governing the agent's behavior. This layer is bounded by the context window's size limit and is entirely volatile. It disappears the instant the process holding it dies. The agent's append-only message log functions as its working memory, and persisting that log makes crash recovery, session resumption, and later debugging possible.
The second layer is external structured memory: the persistent key-value stores or document stores holding facts meant to survive across sessions, things like user preferences, entity attributes, and configuration values. This layer tends to survive a process restart naturally, since it already lives outside the agent's process. The checkpoint still has to record which version of that external data the agent was actually working against at the moment of the snapshot, or recovery can silently reattach the agent to stale facts.
The third layer is tool state, which concerns what the agent did. Cursor positions in paginated APIs, a partially built report, the list of records already processed, the files written, the transformations already applied are distinct from the conversational record, and most checkpoint designs overlook them. Without persisting tool state, a coding agent that has already indexed a repository has no way to know that on its next invocation, and it starts that indexing work over from nothing.
The fourth layer is OS-level effects: filesystem changes, processes the agent spawned, packages it installed, background services it started, and whatever runtime artifacts its tool calls left behind in the execution environment. Purely application-level checkpoint frameworks cannot reach this layer by construction, because they were never built to observe it. Crab's analysis shows that application-level recovery methods preserve conversational history or file state but miss OS-side effects such as installed packages, spawned processes, and files modified by shell commands, so recovery can look correct from the chat transcript while being wrong about what the environment actually contains. Crab's research found that chat-only recovery methods achieve just 8% recovery correctness on these workloads. The underlying cause is a structural mismatch rather than a bug: application frameworks see tool calls but have no visibility into their OS effects, while the OS sees every state change but has no turn-level context to judge which of those changes matter for recovery. No single-layer approach closes that gap on its own. A checkpoint architecture has to treat all four layers as one specification, not four separate problems to solve independently.
Where naive implementations of the canonical checkpoint pattern break
The standard checkpoint-restore pattern is simple enough to state in a single sentence and simple enough to get wrong in several distinct ways when implemented. Before each step, the agent serializes its full state to persistent storage. It then executes the step. On success, it updates the checkpoint with the new state. On crash, the system reloads the last successful checkpoint and resumes from the step after it. Each of those four operations carries its own failure mode.
Frameworks that implement this pattern do most of the bookkeeping automatically. LangGraph's checkpointer system serializes state to a configured backend after every node execution, and any invocation carrying a thread_id resumes from the last checkpoint recorded under that identifier. MemorySaver suits development work, PostgresSaver is the production-grade backend, and SqliteSaver is meant for local or single-process development. Building the same pattern in raw Python, with a state file or a database row standing in for the checkpointer, forces a developer to confront each failure mode directly.
The first failure mode appears when the agent calls a tool but crashes before that tool's result gets checkpointed. Recovery, working from the last good checkpoint, will retry the tool call. That retry is only safe if the tool call is idempotent, and a large share of the tool calls agents make in practice are not: API writes, database inserts, and file uploads all produce duplicate effects when executed twice. Handling this correctly requires either building at-least-once awareness into the tool handler itself or adding a fencing mechanism to the checkpoint record that prevents a replayed call from taking effect twice.
The second failure mode occurs when the agent's state lives inside the same worker process that might crash. If that process dies, the state housed inside it dies along with it, regardless of how carefully the checkpoint logic inside that process was written. The session has to live outside the worker entirely, in a durable store that records model calls, tool results, and any approvals granted along the way, so that a different worker can pick up execution at the last safe point without needing the original process to still exist.
The third failure mode is a matter of granularity: checkpointing on every single turn is correct but expensive, while skipping checkpoints on turns that genuinely produce no recoverable state costs nothing. The trouble is that checkpointing on every turn regardless of whether that turn touched the operating system at all burns a large amount of I/O for no recovery benefit. Crab's analysis of realistic agent workloads found that more than three quarters of agent turns produce no recovery-relevant state. A naive per-turn checkpointing strategy pays that I/O cost on the vast majority of turns for nothing. Solving that problem requires a scheduling approach that can tell, turn by turn, which ones actually need to be checkpointed.
Semantics-aware checkpoint scheduling: capturing only what recovery needs
The right checkpoint granularity for any given agent turn is set by what that turn actually did to the operating environment, not by a fixed schedule applied uniformly across every turn. A runtime capable of classifying those OS-visible effects turn by turn can eliminate most checkpoint work without giving up anything on recovery correctness.
Crab operationalizes that principle with a three-component architecture. An eBPF-based Inspector observes the OS-visible effects that follow each agent turn, including filesystem changes, process state, and memory modifications, without needing to modify either the agent or the checkpoint/restore backend underneath it. A coordinator then aligns checkpoint boundaries with the natural turn boundaries of the agent's execution and overlaps checkpoint I/O with the time the agent spends waiting on the LLM to generate its next response, since the CPU sits largely idle during that wait regardless. A host-scoped engine schedules checkpoint traffic across sandboxes running on the same host, smoothing out bursts and prioritizing the jobs that would otherwise stall the agent's own progress.
The Inspector classifies each turn into one of four checkpoint granularity levels: no checkpoint needed, a filesystem-only checkpoint, a process-only checkpoint, or a full checkpoint. Matching the checkpoint type to the actual effect profile of the turn, rather than defaulting to the heaviest option every time, is what lets the system cut checkpoint traffic sharply while still reaching complete recovery correctness. On shell-intensive and code-repair workloads, Crab raises recovery correctness from 8% under the chat-only baseline to 100%, cuts checkpoint traffic by up to 87%, and stays within 1.9% of fault-free execution time.
None of the backends doing the actual work here are exotic. Crab's v0 backend uses runc for sandbox lifecycle management, CRIU for process state, and ZFS snapshots for filesystem state, all established tools. What is novel is the semantic classification layer sitting above those backends, deciding when to invoke them and at what granularity, rather than any new capability in the backends themselves. The broader lesson applies to developers who aren't running Crab directly: classify the side-effect profile of each step before deciding whether and how to checkpoint it, and schedule checkpoint I/O to overlap with the time the agent spends waiting on model inference, since that wait is a natural, otherwise-wasted window for the work to happen in.
State schema versioning and the event-sourcing alternative
A checkpoint only has value if the agent can actually deserialize it and act on what it contains, which turns schema evolution from a housekeeping chore into an availability concern. A mismatch between the schema a checkpoint was written under and the schema the current agent code expects produces a recovery failure indistinguishable, from the outside, from an outright crash.
That problem appears whenever agent code changes in ways that touch its state shape: new fields added for new tool outputs, existing fields renamed, data types changed. Any one of these can leave a previously valid checkpoint unreadable by the updated agent. Without explicit version tracking, the agent has no way to tell "this checkpoint predates the current schema" apart from "this checkpoint is corrupted," and it will typically treat both cases identically, which is to say badly. Using a Pydantic BaseModel with a strict schema configuration, including extra="forbid", catches schema pollution the moment it enters the system by rejecting illegal fields. That defends against one class of corruption, but it does not substitute for explicit version tracking across schema changes.
Event sourcing offers a different foundation for the same problem. Instead of overwriting a state snapshot each time something changes, the system writes every state change as its own append-only event, and the agent's current state gets derived by replaying that event log from the start. The event log becomes the one source of truth, and recovery becomes a matter of replaying events up to the last confirmed step and continuing from there. Schema migration under this model means replaying old events through a new projection function, rather than trying to migrate already-stored state blobs into a new shape directly.
That approach comes with real costs. Event sourcing is more complex to implement than a snapshot-based checkpoint, and replaying a long event history adds latency for agents that have accumulated a large number of turns. Snapshot-based checkpoints resume faster precisely because they skip the replay step, but they require explicit migration logic whenever the schema changes underneath them. Reducer functions in graph-based frameworks like LangGraph already encode a form of this event-sourcing logic: state updates on annotated fields are treated as merge operations rather than overwrites, and when parallel nodes modify the same annotated field, the reducer resolves the conflict rather than letting the last write simply win, though fields without an annotation still default to last-write-wins behavior. That default makes state evolution traceable and auditable wherever reducers are actually used. The practical recommendation for most production agents is a hybrid: periodic full snapshots paired with an event log recording everything since the last snapshot, which gets the fast resume time of snapshotting together with the schema flexibility that event sourcing provides.
Storage placement and the control-plane boundary for checkpoint state
Checkpoint state stored inside the same execution environment it is meant to protect is not a checkpoint. It functions as a local variable that happens to be written to disk, and it offers no protection against the one failure, the destruction of that environment, that checkpointing exists to survive. Durable recovery depends on checkpoint state living in infrastructure that outlasts the failure it is designed to recover from.
That requirement is what makes the control-plane and execution-plane separation the organizing principle of a sound checkpoint architecture. The control plane handles auth, billing, audit logs, human review, and recovery state, and it lives in trusted infrastructure sitting outside any single container or virtual machine. The execution plane is the sandbox where the agent actually runs: where it installs packages, writes files, and executes commands. That plane is treated as disposable, able to be destroyed and recreated without any loss of recovery state, because none of the state that matters for recovery lives there. This separation makes worker takeover possible. A new worker simply loads the last checkpoint from the control plane and resumes from it, and the environment of the crashed worker becomes irrelevant to that process.
Choosing a storage backend for checkpoint state depends on the workload it needs to serve. In-memory state buffers, the kind LangGraph uses by default in-process, suit development, testing, or single-session tasks where a crash is an acceptable outcome and losing state on restart carries no real cost. These buffers offer no persistence across process restarts and should not be mistaken for a production recovery layer. Database-backed checkpointing, using PostgreSQL in production, SQLite for local development only, or Redis through LangGraph's checkpointer interface, fits multi-step workflows that need to pause and resume reliably, with each checkpoint capturing the agent's full state keyed by thread_id. File-based persistence works well for long-running workflows producing large outputs such as reports, datasets, or media, though local filesystems are not shared across workers, and reaching for cloud object storage instead introduces its own latency and credential management overhead. Key-value stores with a TTL, such as Redis or DynamoDB, suit high-throughput concurrent agents operating within natural expiration windows, though they still need a separate layer to handle file artifacts that don't belong in a key-value record. The storage choice is as much a part of the recovery system as the serialization logic that decides what gets written to it, and has to be settled as the architecture is built.
