ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory

🕒 Published (v1): 2026-07-15 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

General robotic agent OS with lifelong multimodal memory; runtime architecture for agent harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ABot-AgentOS is a Robotic Agent Operating System that interposes a deliberative agent layer between VLM/VLA foundation models and low-level robot controllers, providing scene-conditioned planning, context-isolated skill execution, multi-stage verification, and a persistent multi-modal graph memory. It also introduces EmbodiedWorldBench, an executable benchmark with 200+ tasks across 16 scenes, and a failure-driven self-evolution loop that converts diagnosed memory failures into gated runtime improvements without ground-truth leakage.

Problem

Three structural gaps block long-horizon embodied agents: (1) a reasoning-execution gap — most foundation-model-based robotic systems map model outputs directly to actions, lacking an intermediate layer for task decomposition, skill delegation, and recovery; (2) an embodiment-generalization gap — systems are tightly coupled to specific hardware or control APIs; (3) a persistent embodied-memory gap — short-term buffers and text-only caches cannot preserve multi-modal, relational, source-grounded experience across sessions. A companion evaluation gap exists because no benchmark tests executable multi-scene long-horizon scenarios with dynamic events and trace-grounded scoring.

Method

Agent OS layer. ABot-AgentOS sits above low-level controllers and organizes execution as a closed loop of reasoning, execution, and verification via the Agent Harness, which contains three decoupled roles: - Main LLM (scene-conditioned planner): interprets instructions against current scene state, map, memory, and robot state; forms a revisable plan with explicit completion conditions; delegates multi-step subtasks to the Skill Runner. - Skill Runner: a skill-level subagent with an isolated local context (subgoal, recent observations, failed attempts, recovery strategy); returns a compressed semantic summary to the main LLM, shielding global reasoning from procedural detail. - Verifier: applies supervision at three stages — runtime (stagnation/loop detection), skill-level (semantic objective satisfied?), and finish-time (instruction vs. execution history vs. observed state) — before allowing task termination.

Edge-Cloud Routing. A learned routing policy determines whether a request is handled by an on-device Tiny LLM or escalated to a cloud Large LLM, balancing latency with capability.

Universal Multi-modal Graph Memory (UMGM). Experience is written into a typed, source-grounded graph of nodes (entities, events, places, sessions, visual evidence frames, temporal context) and edges (spatial, temporal, provenance, co-occurrence). A fixed hybrid graph retriever surfaces local subgraphs and records retrieval traces. Separate edge private memory (robot-specific maps, interaction history) and cloud common memory (shareable knowledge) allow cross-robot transfer.

Failure-driven lifelong self-evolution. After each evaluation or deployment split, failure traces are diagnosed across six failure modes (memory writing, evidence selection, frame selection, temporal grounding, entity matching, answer composition); safe fixes are compiled into gated evo-assets promoted only to later splits, preventing current-split ground-truth leakage.

Small-model training pipeline. Teacher trajectory distillation → SFT → online RL with an LLM-as-a-Judge reward → self-evolving reward engine with meta-judge validation and multi-agent self-evolution (details not publicly released).

Key Contributions

• Modular Agent OS architecture decoupling cognition from hardware, supporting heterogeneous embodiments (humanoids, quadrupeds, mobile manipulators) via plugin-based skill integration.
• EmbodiedWorldBench: 16 indoor/outdoor/hybrid scenes, 4 difficulty levels, 200+ tasks spanning navigation, object search, NPC dialogue, dynamic events, and trace-grounded scoring.
• Universal Multi-modal Graph Memory with typed nodes/edges, hybrid retrieval, and inspectable provenance traces.
• Failure-driven self-evolution loop with leakage-safe gated evo-asset promotion.
• Deployment-oriented small-model training pipeline combining teacher distillation, SFT, RL, and meta-judge reward.

Results

• EmbodiedWorldBench (initial subset): ABot-AgentOS improves over single-controller baseline on both task success rate and goal completion (specific numbers not reported beyond directional improvement).
• Memory benchmarks (ABot-AgentOS Static):
• LoCoMo: 87.5
• OpenEQA EM-EQA: 59.9
• Mem-Gallery: 88.6
• NExT-QA Acc@All: 76.5
• After self-evolution:
• LoCoMo: 88.7 (+1.2)
• OpenEQA EM-EQA: 60.4 (+0.5)
• Mem-Gallery: 89.0 (+0.4)

Limitations

• EmbodiedWorldBench agent evaluation results are from an "initial subset" only; full-scale numbers are not reported.
• Small-model training pipeline results (production metrics, private training data) are withheld; only the method is described.
• Self-evolution evo-assets are promoted only to later evaluation splits, so within-split improvement is not possible — latency between failure diagnosis and deployment is inherent.
• The paper does not report ablations isolating the contribution of each Agent Harness component (Skill Runner vs. Verifier vs. memory) on EmbodiedWorldBench, making component-level attribution unclear.
• Memory graph construction depends on VLM extraction quality; errors in entity recognition or spatial grounding at write-time propagate silently into the graph.

Relevance to Agentic AI / LLM Agents

ABot-AgentOS directly operationalizes the core challenges of agentic systems — planning, tool use, memory, verification, and self-improvement — in the physically grounded domain, making it a concrete architecture test-bed for ideas like ReAct, MemGPT, and hierarchical agents under partial observability and actuation uncertainty. The multi-stage Verifier and context-isolated Skill Runner address a known failure mode in LLM agents (language-level belief diverging from world state), which is equally relevant to software and digital agents. The failure-driven self-evolution loop with leakage-safe gating is a methodological contribution applicable to any continual-learning agent system. EmbodiedWorldBench fills a benchmark gap by requiring trace-grounded evaluation of multi-step embodied behavior, complementing existing digital-agent benchmarks like OSWorld.