ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory

🕒 Published (v1): 2026-07-11 15:24 UTC · Source: HuggingFace · link

Why this paper was selected

Robotic agent OS with lifelong multimodal memory; directly relevant to agent harness architecture

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ABot-AgentOS is a robotic Agent Operating System that sits above low-level controllers and VLMs/VLAs, providing deliberative planning, context-isolated skill execution, multi-stage verification, and a persistent multi-modal graph memory. It introduces EmbodiedWorldBench, a 200+ task executable benchmark across 16 indoor/outdoor/hybrid scenes. A failure-driven self-evolution loop converts diagnosed memory failures into gated runtime improvements without leaking current-split ground truth.

Problem

Existing VLM/VLA-based robotic systems face three gaps: (1) a reasoning-execution gap — no intermediate agent layer between model outputs and physical actions for decomposition, tool use, verification, and recovery; (2) an embodiment-generalization gap — tight coupling to specific hardware/APIs prevents cross-platform transfer; (3) a persistent embodied-memory gap — text-only caches and short-term buffers cannot preserve multi-modal, relational, source-grounded experience across long-horizon interaction. No benchmark covers executable multi-scene, multi-event long-horizon evaluation with trace-grounded scoring.

Method

ABot-AgentOS comprises three tightly integrated layers:

Agent Harness. A hierarchical LLM framework with three separated roles: (a) a main LLM performing scene-conditioned task planning (instruction + map + memory + robot state → revisable plan with explicit completion conditions); (b) a Skill Runner subagent handling procedural subtasks in an isolated local context, absorbing low-level collision/recovery detail and returning compressed semantic outcomes; (c) a Verifier applying three-stage supervision — runtime (trajectory stagnation/loops), skill-level (semantic objective satisfaction), and finish-time (full consistency check before termination). Edge-cloud routing routes routine turns to an on-device Tiny LLM and escalates to a cloud Large LLM based on a learned routing policy.

Universal Multi-modal Graph Memory (UMGM). Experience is converted into a typed graph of nodes (entities, events, places, sessions, visual frames, temporal anchors) and edges (spatial, temporal, causal, provenance). Retrieval uses a fixed hybrid graph retriever that returns local evidence subgraphs with traceable paths rather than raw text or full video. Memory is split into edge-side private memory (per-robot maps, preferences, local experience) and cloud-side common memory (shareable maps, general task traces) with upload/download for cross-robot transfer.

Failure-Driven Lifelong Self-Evolution. After each evaluation or deployment split, the system diagnoses failures across six categories (memory-writing, evidence-selection, frame-selection, temporal-grounding, entity-matching, answer-composition) and compiles fixes into gated evo-assets that are promoted only to later evaluation splits, preventing ground-truth leakage. A multi-agent self-evolution loop is paired with an LLM-as-a-Judge reward engine and Meta-Judge validation for small-model training via SFT + RL.

EmbodiedWorldBench. 16 scenes (indoor/outdoor/hybrid), 4 difficulty levels, 200+ tasks spanning navigation, object search, NPC dialogue, and dynamic events. Scoring is trace-grounded.

Key Contributions

• Modular robotic Agent OS architecture decoupling high-level VLM reasoning from low-level controllers, supporting heterogeneous embodiments (humanoids, quadrupeds, manipulators) via plugin-based skill interfaces
• Three-role Agent Harness (planner, Skill Runner with context isolation, multi-stage Verifier) closing the reasoning-execution gap
• Universal Multi-modal Graph Memory encoding dialogue, egocentric visuals, object states, spatial/temporal relations, and task traces as typed, source-grounded graph records
• Failure-driven lifelong self-evolution loop with gated evo-asset promotion, preventing current-split leakage while enabling continual improvement
• EmbodiedWorldBench: first executable long-horizon embodied benchmark with dynamic events, NPC interaction, and trace-grounded scoring across 16 multi-type scenes
• Edge-cloud collaborative architecture with learned routing and private/shared memory hierarchy for deployment-feasible operation

Results

EmbodiedWorldBench (agent evaluation): - ABot-AgentOS improves over a single-controller baseline on both task success rate and goal completion rate (exact delta figures not provided in the supplied text)

Memory benchmarks (ABot-AgentOS Static): - LoCoMo: 87.5 - OpenEQA EM-EQA: 59.9 - Mem-Gallery: 88.6 - NExT-QA Acc@All: 76.5

After self-evolution: - LoCoMo: 88.7 (+1.2 over static) - OpenEQA EM-EQA: 60.4 (+0.5 over static) - Mem-Gallery: 89.0 (+0.4 over static)

Limitations

• Full numeric comparisons against prior systems on EmbodiedWorldBench are not disclosed in the available text; only improvement direction over a single-controller baseline is stated
• Small-model training pipeline targets business deployment; private training data and production results are not released, limiting reproducibility of that component
• Self-evolution gains are modest (+0.4–1.2 points), suggesting diminishing returns from the current failure-diagnosis mechanism
• UMGM graph construction relies on VLM extraction quality; errors in entity/relation extraction at write-time propagate silently into retrieval
• EmbodiedWorldBench is newly introduced and not yet widely adopted, so cross-system comparisons with the broader community are absent
• Edge-cloud routing policy is learned from internal data; generalization to novel task distributions or new embodiments is not characterized

Relevance to Vision-Language Models

ABot-AgentOS directly addresses how VLMs can be operationalized for long-horizon physical tasks — not by fine-tuning them for every robot, but by layering a deliberative Agent OS above them that handles planning, memory, and verification. The multi-modal graph memory is architecturally significant for VLM researchers because it decouples episodic grounding from the VLM's context window, enabling persistent, auditable visual evidence binding (egocentric frames, object states, spatial relations) that raw prompt stuffing cannot achieve. The failure-driven self-evolution loop, using LLM-as-a-Judge with Meta-Judge validation, offers a template for continual adaptation of VLM-backed agents without retraining the backbone. EmbodiedWorldBench provides a more demanding evaluation surface for VLMs than static VQA, requiring multi-step grounded reasoning under partial observability and dynamic events.