Crab: A Semantics-Aware Checkpoint/Restore Runtime for Agent Sandboxes¶

🕒 Published (v1): 2026-04-30 17:20 UTC · Source: Arxiv · link

Why this paper was selected

Checkpoint/restore runtime for agent sandboxes; enables rollout branching and fault tolerance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Crab is a host-side runtime that adds semantics-aware checkpoint/restore (C/R) to agent sandboxes (containers/microVMs) by bridging the agent–OS semantic gap: an eBPF inspector classifies per-turn OS-visible effects to decide checkpoint granularity, a coordinator overlaps checkpoint work with LLM wait time, and a host-scoped engine coordinates I/O across co-located sandboxes. On Terminal-Bench and SWE-Bench, Crab achieves 100% recovery correctness while staying within 1.9% of fault-free execution time and cutting checkpoint traffic by up to 87%.

Problem¶

Existing C/R approaches for sandboxed agents fall into two irreconcilable extremes. Application/framework-level recovery (e.g., LangGraph, Claude Code's chat+file persistence) preserves conversational state but misses OS-side effects—installed packages, spawned processes, runtime memory—yielding only 6–48% task success after failures. Full per-turn OS/VM checkpointing (e.g., E2B/Firecracker) is correct but imposes \(\geq\)4 s overhead per checkpoint; at 100 co-located sandboxes with median turn time of 3.34 s, this overwhelms host I/O (up to 3.78× slowdown). The root cause is the agent–OS semantic gap: agent frameworks see tool invocations but not their concrete OS effects; the OS observes state changes but lacks turn-level context to judge recovery relevance. Over 75% of agent turns produce no recovery-relevant state, yet this sparsity is invisible to either layer alone.

Method¶

Crab introduces three cooperating components, all transparent to agents and C/R backends:

Coordinator — an HTTP reverse proxy interposed on the agent→LLM path. It detects turn boundaries by intercepting outbound LLM requests and inbound responses. After each turn, it queries the Inspector, issues a checkpoint job if warranted, then gates the next turn until the checkpoint is durable—but only blocks if checkpoint work outlasts the LLM wait window (asynchronous overlap, R2). If the LLM response returns before checkpointing finishes, it signals the C/R Engine to promote that job to high priority.

Inspector — a user-space daemon backed by eBPF probes attached to the kernel. After each turn it evaluates net OS-visible changes since the last checkpoint: filesystem inode modifications, process creation/exit, and memory-mapped state. It classifies the turn into one of four checkpoint needs: none, filesystem-only, process-only, or full (filesystem+process). The "net change" semantics (e.g., a file created and deleted within one turn counts as unchanged) are key to eliminating spurious checkpoints.

C/R Engine — a host-scoped scheduler with a worker pool and versioned checkpoint manager. It uses commodity backends: runc for container lifecycle, ZFS snapshots for filesystem state (sub-22 ms even under concurrency), and CRIU for process state. The scheduler smooths bursty checkpoint arrivals across co-located sandboxes, prioritizing jobs that are on the critical path (signaled by the Coordinator) over those still hidden behind LLM wait windows.

Workflow: at turn \(i\) end, the Coordinator intercepts the outgoing LLM request, queries Inspector, conditionally submits a checkpoint job, and forwards the request. When the LLM response arrives, it either releases immediately (checkpoint done) or blocks briefly until done, then starts turn \(i+1\).

Key Contributions¶

Identification and quantification of the agent–OS semantic gap: chat-only recovery succeeds on only 8–13% of Terminal-Bench tasks; every-turn full checkpointing at 96-sandbox density adds up to 3.78× slowdown.
Design and implementation of Crab: a transparent host-side runtime combining eBPF-based turn-effect classification, asynchronous LLM-wait-window overlap, and host-scoped C/R scheduling—without modifying agents or C/R backends.
Demonstration that Crab achieves 100% recovery correctness on both Terminal-Bench (shell-intensive) and SWE-Bench (code-repair) while keeping exposed checkpoint overhead within 1.9% of fault-free execution time at up to 96-sandbox density.
Agent-facing rollback API case study: exposing C/R as a tool reduces wall-clock time by up to 29% and rollback tokens by 36% versus shell-level self-recovery.

Results¶

Recovery correctness: Crab 100% vs. chat-only 6–28%, chat+FS 34–48%, on Terminal-Bench (50 tasks, one injected failure each).
Fault recovery overhead: within 1.9% of no-fault, checkpoint-free baseline; restart-from-scratch adds up to 1.67× on SWE-Bench.
Checkpoint traffic reduction: up to 87% of turns classified as requiring no checkpoint, eliminating corresponding C/R work.
Co-location scalability: every-turn full checkpointing at 96-sandbox density adds up to 3.78×; Crab p95 exposed checkpoint delay is 0.44% of task time at 64 co-located sandboxes.
Rollback tool case study: replaces brittle shell-level self-recovery, saving up to 29% wall-clock time and 36% rollback tokens.
Baselines: Claude Code (chat+git), LangGraph (conversation), Docker (full container), E2B/Firecracker (full VM), restart-from-scratch.

Limitations¶

The Inspector's eBPF approach requires Linux kernel support and privilege; portability to non-Linux sandboxes (e.g., Windows containers, macOS VMs) is unaddressed.
CRIU process checkpointing remains expensive for large memory footprints (e.g., 64 concurrent 1 GB dumps take 47 s); Crab mitigates but does not eliminate this via scheduling and selective checkpointing.
The Coordinator requires interposing on the agent–LLM HTTP path; agents that use non-HTTP or encrypted/authenticated transports may need adaptation.
Evaluation is limited to two benchmarks (Terminal-Bench, SWE-Bench) and three agents/LLMs; generalization to other agent architectures or workload distributions is not demonstrated.
The agent-facing rollback API requires tool-call integration; agents not designed to use rollback gain no benefit from the proactive C/R case study.

Relevance to Harnesses / Meta-Harnesses¶

Crab is directly relevant as a systems substrate layer beneath any agent harness that runs tools in sandboxed environments: it provides the C/R primitives—fault tolerance, spot preemption, RL rollout branching, and rollback—that meta-harnesses need to orchestrate long-horizon, multi-agent, or post-training workloads reliably. The RL rollout use case (Tree GRPO branching from intermediate sandbox states) is particularly important for harnesses that drive agents through tree-search or MCTS-style exploration without re-executing shared prefixes. Exposing rollback as an agent-callable tool also lets harness designers delegate error recovery from fragile shell scripts to a single C/R primitive, simplifying harness control logic. For researchers building meta-harnesses that co-locate many agent instances (e.g., parallel evaluation or RL rollout fleets), Crab's host-scoped scheduler directly addresses the I/O contention problem that would otherwise constrain sandbox density.