File System Persistence Patterns for Long-Running Agents

Anthropic's engineers describe the failure mode: picture a software project staffed by engineers working in shifts, where each new engineer arrives with no memory of what happened on the previous shift. That's the condition every long-running agent operates under by default, and it's why session changes without a disciplined handoff produce a collapse in output quality rather than a graceful continuation. The model itself hasn't degraded between sessions. The workspace it wakes up into has, because nobody designed the handoff.

The collapse in output quality between sessions is almost never a model failure; it is a workspace design failure. The fix runs across four interlocking pieces: a directory layout that separates goals, work-in-progress, completed artifacts, and decision logs, so a fresh session can reconstruct the minimum needed state without a broad scan; artifact conventions that stamp every file with a status field, a source field, and a session ID, so the agent can distinguish a verified conclusion from a stale draft or ingested external content; checkpoints written after every irreversible action and before every subagent handoff, capturing side-effecting steps as completed so they don't execute twice on resumption; and a recovery hook that runs before any task work begins, reading the goal file, loading the last checkpoint, and rejecting any artifact with unknown provenance before it enters the active context. Underlying all of it is the requirement that the file system actually persist, whether through persistent disk volumes or a snapshot-restore model that preserves the full execution state rather than tearing the environment down between runs.

Addy Osmani's 2026 analysis names three walls that every long-running agent eventually hits: finite context, no persistent state, and no capacity for self-verification. None of these are solved by a smarter model or a longer context window. They're solved, to the extent they get solved at all, by the file system, which functions as the primary tool for addressing all three constraints outside the model's own weights.

What the file system does in a multi-session agent

Large language models are stateless by design. Every call starts from scratch, with nothing carried over natively from the last one. Early agent builders tried to route around this by stuffing the entire conversation history into the context window on every call. That approach breaks down fast: latency spikes, relevant facts get buried under irrelevant ones, and when two versions of the same fact both sit in the window, the model has no principled way to pick the current one. Bigger context windows just treat the file system as deliberate architecture rather than an afterthought bolted onto a stateless model.

Memory in an agent system does three jobs: episodic memory tracks what happened, semantic memory holds facts about the domain, and procedural memory encodes how tasks should be done. The file system is where all three get durably expressed between sessions, because nothing durable survives in the model's own context once the call ends. Underneath that, a memory-state cycle repeats on every task: the agent reads from memory to build its initial state, updates that state as work proceeds, and at the end writes select pieces back to memory for the next session to find. The file system is the physical medium that cycle runs on.

A deep agent architecture sits on four pillars: a planning tool that keeps goals in recent attention, a virtual file system that offloads context out of the prompt window, subagent delegation with per-subagent context isolation, and long-term memory that persists across runs; LangChain formalized this in its deepagents library, first published to PyPI on July 29, 2025, after Claude Code, Deep Research, and Manus proved it in production. A subagent can write its findings to a path like research/competitor-a.md, and the orchestrator reads back a clean summary without ever pulling the subagent's noisy exploration into its own context. Store state outside the prompt as artifacts, decisions, and tool outputs, then pull back only what the current loop actually needs.

State and memory get conflated constantly, and the conflation causes real design failures. State is a snapshot, everything the agent currently knows about a task right now: what step it's on, what the last tool call returned, what it's tracking. It behaves like a whiteboard, updated constantly during the task and erased when the session ends unless something deliberately persists it. Memory is the separate mechanism that carries information across that session boundary. Broken state makes an agent lose track mid-task. Broken memory means it can't learn anything across sessions. Both failures occur constantly in production, and each needs its own fix; treating them as one problem guarantees neither gets solved.

Workspace layout: structuring directories for a fresh session

A workspace layout functions as the schema an agent uses to locate its own state after waking into a session with no other context to draw on. Get the layout wrong, and the virtual file system stops offloading context out of the prompt window, because the agent has to scan broadly just to figure out where anything lives.

Subagent isolation depends on paths that encode scope. A subagent writing to research/competitor-a.md is legible to the orchestrator precisely because the path itself tells the orchestrator what subagent produced it and what kind of artifact it is.

A workable layout separates at least four kinds of content:

/workspace
 /goals <- current task, plan, acceptance criteria
 /wip <- in-progress, unverified artifacts
 /done <- completed, verified artifacts
 /logs <- decision records, session history

Goals and plans capture what the agent is trying to do. Work-in-progress artifacts hold what it has produced so far but hasn't verified. Completed artifacts hold what it considers finished. Logs record why it made the choices it made. These four map directly onto the episodic, semantic, and procedural memory types described above, giving the layout a conceptual backbone rather than an arbitrary folder scheme.

The goals path deserves special placement: it should sit in a location the agent checks first on every session start, since keeping goals in recent attention is one of the four pillars of the deep agent architecture. The file system is the primary tool for solving the three walls every long-running agent hits outside the model: finite context, no persistent state, and no self-verification.

A reasonable objection follows naturally here: why not just use a database instead of a file system? Databases suit semantic memory well, vector retrieval and structured fact storage in particular, but the file system is the right layer for artifacts and workflow state that need to be readable by the agent's own tool calls, auditable by a human who wants to check the agent's work, and portable across whatever infrastructure the agent happens to run on. The two layers complement each other rather than competing for the same job.

Artifact conventions: naming, format, and provenance

An agent that can't tell a completed artifact from a draft, or its own reasoning from something it absorbed off an external feed, is set up to act on stale or corrupted state without any way to notice. That's the practical stake behind conventions that otherwise look like pedantic file-naming rules.

Artifacts should carry metadata the agent can read without parsing the full content: at minimum a status field marking it draft, verified, or superseded, a source field distinguishing self-generated output from externally ingested content, and a session ID or timestamp identifying which run produced it. The source field carries more architectural weight than it looks like it should. Research on the HEARTBEAT vulnerability shows that content ingested during an agent's background execution enters the same memory context as its user-facing conversation. Without a source field on every artifact, the agent has no way to distinguish a conclusion it reasoned its way to from a claim it picked up off a monitored feed.

Structured formats, JSON, YAML frontmatter, Markdown with defined sections, let the agent pull a specific field without re-reading an entire document. Flat prose artifacts force full re-ingestion every time, and the context load grows with every session that touches them.

Procedural memory raises its own version of this problem. Workflow templates and task patterns stored as artifacts need a version field, because an agent that reads a superseded workflow and executes it faithfully hasn't made a reasoning error. It has run into a file system design failure that gave it no way to know the workflow was out of date.

Naming conventions carry the same weight as metadata fields. Encoding an artifact's scope and status directly into its path, not buried in its content, is what keeps subagent isolation legible to an orchestrator that doesn't have time to open every file it finds.

How undisciplined memory ingestion corrupts agent behavior

The deeper architectural risk in persistent file-based memory is silent pollution. It's silent pollution: external content absorbed during background execution gets promoted into durable memory without the agent or the user ever noticing it happened.

The HEARTBEAT research, conducted at NTU and published in March 2026 based on an OpenClaw release from late January 2026, documents exactly this mechanism. Heartbeat-driven background execution runs inside the same session as user-facing conversation, so content from monitored email, message channels, news feeds, code repositories, and social platforms enters the identical memory context the agent uses for its foreground interactions with a user.

The paper traces a pathway it calls Exposure to Memory to Behavior: misinformation encountered during background execution enters short-term session context, gets written into long-term memory through the agent's own ordinary save behavior, and later shapes what the agent does in a session that has nothing to do with the original exposure. No prompt injection required. The agent doesn't get tricked by an adversarial instruction; it absorbs something that looked credible in the moment and later treats it as established fact.

Under naturalistic browsing conditions, routine memory-saving behavior promoted short-term pollution into durable long-term memory at rates up to 91%, and the resulting behavioral influence persisted across sessions even once the original polluting content had dropped out of the active context. Social credibility cues, especially the appearance of consensus, drove most of the short-term influence measured in the study. The content just needs to look like something other people already agree on.

The OWASP Top 10 for Agentic Applications, published in December 2025, gives this failure mode a formal home, covering goal hijacking, tool misuse, identity and privilege abuse, cascading failures, and rogue agent behavior. Prompt injection is a category in the separate OWASP LLM Top 10, not in this agentic taxonomy.

The response lives at the file system level: source provenance on every artifact, a write policy that keeps self-generated conclusions clearly separated from ingested content, and a checkpoint review step before anything ingested gets committed to long-term storage. None of that is security theater. Each piece directly interrupts the exposure-to-memory-to-behavior pathway the research describes.

Checkpoint discipline: exact resumption, not backup

Checkpointing makes the next session's starting state exact rather than merely approximate.

Graph-based agent frameworks model a workflow as nodes and edges, and after each step, the framework persists the workflow's state, its variables, its history, its current position, to a durable store such as PostgreSQL or SQLite. If the agent crashes, it reloads the last checkpoint and resumes from there rather than starting over.

Resumption is not the same guarantee as exactly-once execution, and conflating the two causes real bugs. If a node had partially executed before the crash, sent an email, written a database row, that action can fire again once the agent resumes from the last checkpoint. A checkpoint that captures "email sent" as a completed step protects against duplication. A checkpoint that captures the node as merely "in progress" does not, and the agent will send the email twice. Side-effecting steps need to be built idempotent from the start, not patched to be idempotent after the first duplicate email goes out.

Some things can't be checkpointed at all: open file handles and live client objects don't survive a serialization boundary. That constraint should shape workspace layout directly, since persistent disk artifacts need to stand in for in-memory handles as the durable reference an agent falls back on.

Timing matters as much as content. Checkpoint after every irreversible action, not only when a crash happens to occur, before any handoff to a subagent, and before any external write. Each checkpoint functions as the agent's contract with itself that resuming won't mean re-running work already done.

Osmani's context-rot finding sharpens the case for early, frequent checkpointing: model performance degrades as the context window fills, well before the window's hard limit is even reached. Checkpointing lets a fresh session start from a clean context built off the checkpoint rather than replaying the full history from scratch. One caution applies here: as a working-memory buffer approaches its token limit, a summarization process compresses older turns into a dense background summary, but mid-conversation summarization rewrites the prompt prefix and invalidates the KV cache, producing a latency spike on the very next call. Checkpoints timed to coincide with natural summarization boundaries avoid paying that cost twice.

How does a recovery hook reconstruct agent state at session startup?

Session startup is the most neglected part of long-running agent design, and it's where every file system decision made upstream either pays off or falls apart.

A recovery hook is a structured startup routine the agent runs before touching any task: read the goal file, load the last checkpoint, verify artifact status fields, and reconstruct the minimum state needed to continue rather than the entire history. Reconstructing only the minimum state rather than the entire history keeps the agent rebuilding only what the current loop actually needs, pulling state from outside the prompt and retrieving back just enough to act, rather than replaying every session that came before it.

Workspace layout is what makes this fast rather than slow. A consistent top-level goal path and artifacts carrying status metadata make the startup scan deterministic and quick. An unstructured workspace forces the agent to read broadly on every wake-up, and that broad read bloats the initial context before any real work has begun.

Provenance checking belongs at this exact point in the sequence. The startup hook should reject or flag any artifact with unknown source provenance before loading it into the active context, and that check is the file-system-level defense against the exposure-to-memory-to-behavior pollution pathway documented in the HEARTBEAT research. A recovery hook that skips this step reopens the door the provenance fields were designed to close.

One more gap deserves attention here. Models reliably skew positive when grading their own work, so the recovery hook is the right place to insert an external verification step: read the last checkpoint, check the artifact's status field, and confirm that anything marked "done" actually meets its acceptance criteria before the task counts as resumed rather than restarted from zero.

The shift-change analogy from the opening closes here. A well-built recovery hook is the briefing document an incoming engineer reads before touching the codebase. When it's accurate and current, the shift change is seamless. When it's missing or stale, the new session burns its first hours re-learning what the last one already knew, and the productivity collapse Anthropic's framing warned about at the start plays out exactly as described.

Infrastructure: persistent disks, VM isolation, and snapshot-restore

Every pattern described so far assumes one thing: that the file system actually persists. Stateless or ephemeral runtimes invalidate all of it silently, because a workspace layout, artifact conventions, and checkpoint discipline mean nothing if the disk they live on disappears between sessions.

The snapshot-restore model addresses this directly by pausing a sandbox with its full execution state intact, reasoning chain, working memory, tool history, and any pending action, rather than tearing the environment down between runs. Resuming a paused sandbox takes about one second, which lets real-world time pass without burning compute cost, and then picks back up with sub-second latency once work resumes. File system state is part of what gets preserved in that pause, not a separate concern handled by a different system.

At the infrastructure level, a Firecracker snapshot captures the guest's entire physical RAM on disk, so the running state is never discarded or rebuilt from a cold start, and that guarantee is what everything above depends on. Maritime, a cloud infrastructure platform that gives each AI agent its own persistent Firecracker micro-VM rather than a shared container or serverless slot, is built around exactly that guarantee. A workspace layout, a set of artifact conventions, and a checkpoint protocol are only as durable as the disk and the runtime that hold them between the sessions an agent is asked to bridge.