Cold Start Latency Benchmarks Across Agent Hosting Approaches

Container image size and runtime choice are the primary levers for Lambda cold starts. Adding an AI agent's dependency stack on top of the same runtime makes the number stop meaning anything useful. Cold start latency for agent workloads is a compounding sum of independent stages, not a single infrastructure metric, and treating it as one makes benchmark comparisons misleading. Several stages stack on top of each other before an agent produces a single token: pulling the container image, initializing the SDKs and their dependency trees, loading model weights where local inference is involved, and initializing the GPU context. The ratio between a cold path and a warm path on the same workload has been measured at roughly 1,000×: a cold-start GPU instance takes many seconds to deliver a first token, but a warm instance responds almost instantly. For a standard web API, that penalty happens once per deployment, and thousands of requests amortize it. Agents provision per task and per retry, not once per deployment, so every layer of cold start tax multiplies across every unsupervised turn an agent takes.

How each cold start layer behaves with and without local inference

The first diagnostic question for any agent deployment is whether it calls a hosted model API or runs inference locally, because the cold start profile for each case is different enough that they need separate mental models.

For an API-calling agent, which is the more common case, the dominant costs are container image size, runtime choice, and where SDK initialization happens in the code. A second, separate lever sits in where SDK clients get initialized: setting them up inside the handler function re-runs the TLS handshake and connection pool setup on every cold start, while moving that same initialization to module scope pays the cost once and lets every subsequent warm invocation skip it.

For an agent running local inference, a different stage dominates the picture. Weight loading alone can exceed 110 seconds for very large models, at which point container pull and SDK initialization become rounding errors by comparison. On top of weight loading sits GPU context initialization, which covers CUDA context creation, memory pool allocation, and kernel compilation, adding further seconds before the model can run at all. Lambda's Firecracker architecture does not support PCIe passthrough, which makes local GPU inference on standard serverless impossible no matter how the code is optimized. That single fact removes an entire category of hosting options for any team running local inference, regardless of what the rest of the benchmark says.

Locating a given workload on this spectrum, API-calling versus local inference, has to happen before any hosting comparison makes sense, because the two cases are optimized by entirely different levers.

The cold start tax from framework and language runtime choice

Before a team ever picks a hosting platform, the agent framework and language runtime it's built on already set a cold start floor that compounds with whatever infrastructure gets chosen later.

CrewAI stands out as a clear case study here: a 2026 test found it meaningfully slower than the rest of the field, with a cold start on first launch running into the tens of seconds, and its install footprint drags in dependencies, including the Kubernetes client and the ChromaDB vector database, that have no connection to agent logic at all. Those packages get imported and initialized regardless of whether the agent ever touches a Kubernetes cluster or a vector store, and the time to import them is paid on every cold start.

Once a model call actually fires, the network round-trip to the LLM provider dominates tail latency no matter which framework is running. Median latencies across frameworks cluster close together for this reason, but outliers like GraphBit and LangGraph show additional orchestration overhead layered on top of the model call itself. Framework choice shapes the cost before the network call and adds variance after it, with the model call itself sitting largely outside any framework's control.

Memory footprint tells a similar story from a different angle. Rust-native frameworks peak at roughly 1 GB per agent instance, while Python frameworks peak at roughly five times that figure, and that gap becomes a scaling constraint on fleet density before it is visible as a latency number on a dashboard.

The practical consequence is that these two layers, framework and infrastructure, don't cancel each other out. A team running a heavy Python framework on a fast snapshot-restore platform is still paying the full framework tax on top of whatever speed the infrastructure buys back. Framework choice and hosting choice are two separate bills, and paying down one doesn't reduce the other.

The latency and isolation trade-offs across the main hosting architectures: serverless functions, containers, and micro-VMs

With the layer-by-layer stack established, the three dominant hosting architectures can be read as three different bets about which of those layers to absorb, eliminate, or defer, and each bet carries consequences that no amount of tuning removes.

Serverless functions in the Lambda style make their bet on cost efficiency at low and bursty traffic, where per-invocation pricing beats paying for idle capacity, at the price of tail latency and any workload that needs a GPU. Direct comparisons on identical ML serving workloads show container deployments on ECS Fargate, Cloud Run, or EKS delivering meaningfully lower mean and P99 latency than Lambda. Serverless functions also carry duration limits that can run too short for an agent task involving multiple LLM calls and tool loops strung together in sequence.

Containerized deployments on Docker or Kubernetes remove the duration ceiling entirely and support GPU instances, which raises the architectural ceiling for what the workload can do once it's running. Cold start for containers is dominated by the same image-pull problem serverless never escapes, 8 to 15 GB for containers, but containers have more room to work around it through warm-pool strategies. That headroom isn't free: it comes with real operational overhead and a minimum idle cost that gets paid even at zero traffic. Containers also share a kernel across workloads, so isolation is enforced through namespaces and cgroups rather than hardware, a distinction that matters directly for any agent executing untrusted or LLM-generated code. Some hosting platforms, such as Maritime, a cloud infrastructure provider that runs each AI agent inside its own dedicated Firecracker micro-VM with a separate kernel, are built from the ground up around that hardware boundary rather than treating it as an add-on.

Micro-VMs built on Firecracker make a different bet again. The isolation boundary there is enforced in hardware and sits below the layer an agent can reason about at all, so a guest-kernel exploit stays contained inside the VM rather than reaching the host. By 2026, practitioner consensus has settled on micro-VM isolation, or an equivalent kernel-level sandbox like gVisor, as the minimum viable boundary for agents running arbitrary or LLM-generated code, with hardened containers serving as an inner layer rather than the outer one. The trade-off is that micro-VMs demand more infrastructure to build and operate, and that added cost buys correct isolation along with the foundation for the snapshot-restore technique covered next.

None of these three architectures is simply faster than the others in the abstract. Each one is built around a different assumption about which cold start layer matters most for the workload running on it, and the right choice depends on which layer dominates the workload under consideration, as laid out in the two sections above.

How snapshot-restore changes the cold start equation for micro-VMs

Snapshot-restore doesn't make a cold boot faster. It replaces the boot entirely, collapsing the full initialization stack into a memory restore that runs in milliseconds rather than seconds.

The mechanism works by keeping a pool of VMs brought to a ready state, with a memory snapshot taken at that point. An incoming request restores from that snapshot instead of booting a kernel from scratch, so every initialization layer, SDK setup, dependency import, connection pool creation, gets paid once at snapshot time rather than on every single invocation. The cost of a cold boot still exists, but it moves to a single point in the system's life rather than recurring on every request.

Production numbers back this up. One deployment, PandaStack, uses snapshot-restore on every create and keeps no warm pool of idle VMs at all, and the restore step comes in fast, keeping end-to-end create latency low at both the median and the p99. A true cold boot in that system only happens on the very first spawn of a given template; every create after that pays restore prices rather than boot prices.

The ecosystem around this technique is also closing a developer-experience gap that had kept it from reaching hot paths. SmolVM, launched April 17, 2026, packages a micro-VM into a single static executable that spawns as easily as a subprocess, from any developer machine on macOS, Linux, or Windows, and runs fast enough to sit on a hot path rather than being reserved for occasional cold starts. Before tools like this existed, Firecracker-based sandboxing was often too operationally heavy to justify using on every request, so teams were pushed toward skipping sandboxing altogether on the paths that mattered most.

That gap matters because of what's actually running inside these sandboxes. Cold starts long enough to be painful tend to push teams toward skipping sandboxing on hot paths entirely, and for AI-generated code, where the very instruction being executed can be shaped by an attacker through prompt injection, that's the wrong trade to make. Snapshot-restore removes the excuse: if the isolation boundary costs milliseconds instead of seconds, there's no latency argument left for leaving code unsandboxed.

Idle cost and cold start decisions at agent fleet scale

Cold start and idle cost are two faces of the same architectural constraint, and pushing hard on one by brute force makes the other worse. Keeping infrastructure always warm removes cold starts but means paying for idle compute around the clock. Scaling fully to zero removes that idle bill but reintroduces a cold start on every request that arrives after a quiet period. Snapshot-restore scheduling is the mechanism that breaks this trade-off rather than simply picking a side of it.

The scale of the underlying waste is well documented. The Aries research framework, published on arXiv in 2026, found that agent tool sandboxes spend most of their time at low utilization, with CPU idle for more than 80% of execution time, punctuated by sharp spikes during the brief windows when a tool actually executes. That pattern creates a genuine dilemma for anyone running an agent fleet: tearing a sandbox down after every tool call cuts idle cost substantially but drives up total cost through repeated snapshot create and restore cycles, while keeping sandboxes persistent eliminates restore overhead entirely but pays for idle compute without interruption. The Aries analysis found that an ideal scheduler can meaningfully cut total cost against the persistent-sandbox baseline, and that gap is large enough to matter at fleet scale.

The hibernate-wake pattern puts scheduling logic into practice: sandboxes hibernate during the 80%-plus of execution time they spend idle and wake through a snapshot restore in the milliseconds before the next tool call, rather than staying fully booted the entire time or being torn down and rebuilt from nothing on every call. That's the structural answer to a problem that neither always-on infrastructure nor pure scale-to-zero can solve on its own: the fleet pays restore prices instead of either boot prices or a continuous idle bill.