Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy

🕒 Published (v1): 2026-06-25 16:36 UTC · Source: Arxiv · link

Why this paper was selected

Unified cyber+physical orchestration for embodied agents; recovery from physical failures

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OmniAct is a hierarchical embodied agent framework that unifies cyber (APIs, IoT) and physical (manipulation, navigation) action spaces under a single planner, adds event-boundary-driven memory compression for sub-linear context growth, and closes the execution loop via asynchronous VLM-based visual preemption. Evaluated on 40 real-world tasks across two robot platforms, it consistently outperforms VLA and LLM-planning baselines while keeping per-call token consumption nearly flat over 270+ interaction rounds. It elevates mid-scale open-weight models to proprietary-level performance without fine-tuning.

Problem

Persistent embodied agents in unstructured environments face three compounding gaps: (1) no unified action space bridges cyber tools (APIs, IoT) and physical control, forcing brittle hand-crafted inter-domain interfaces; (2) linear context accumulation in existing agent frameworks causes memory overflow and semantic conflict between long-term preferences and recent state; (3) VLA policies are inherently open-loop — undetected mid-execution failures cascade into unrecoverable faults.

Method

OmniAct decomposes the agent into three cooperating modules:

Multimodal Semantic Planner. Given multimodal observation \(o_t = \langle v_t, s_t \rangle\) and compressed history \(H_t\), a planner \(\pi\) produces skill-routing actions \(a_t = \langle \tau_t, g_t \rangle\) where \(\tau_t \in \mathcal{T}\) selects a skill and \(g_t\) provides conditioning parameters. A single structured JSON dispatch supports parallel multi-skill invocation within one step: $\(a_t \sim \pi(\cdot \mid c_t, H_t, \mathcal{T})\)$

Asynchronous Visual Preemption Engine. A low-frequency asynchronous monitor periodically samples visual frames and queries a VLM for semantic-level assessment (object grasped? robot progressing?) concurrent with VLA execution. On anomaly detection, execution halts immediately, failure context is packaged, and control returns to the planner for replanning — without relying on force sensing or geometric constraints.

Adaptive Hierarchical Memory. Three tiers: (i) Sensory — FIFO frame buffer for the control policy; (ii) Episodic — event-boundary-driven compression that condenses raw observation sequences into timestamped keyframe summaries only at semantic boundaries (sub-task completion, state change), yielding context that scales with distinct events rather than raw turns; (iii) Reflective — dynamically accumulates structured failure records (root cause + corrective strategy), retrieved at planning time to prevent recurrence and resolve long-horizon preference conflicts.

Key Contributions

• Unified cyber-physical skill-routing action space spanning IoT, web APIs, robotic manipulation, and mobile navigation within a single event-driven planning loop.
• Event-boundary-driven episodic compression achieving sub-linear context growth; per-call tokens remain in the 4k–8k band over 270 rounds vs. linear growth exceeding 120k tokens for raw history.
• Asynchronous visual preemption for closed-loop failure detection and replanning without force sensing, decoupled from low-level VLA execution.
• Reflective memory layer that accumulates causal failure analysis, boosting long-horizon constraint satisfaction by 13–20 points over episodic-only compression.
• Real-world validation: 40 tasks on UR5e manipulator + Keenon mobile robot coordinating 4 IoT devices and 12 virtual APIs; cross-model generalization across 6 backbone families without fine-tuning.

Results

• L3 (Full Multimodal) E2E success: 50.0% manipulation, 54.2% navigation vs. best baselines of 43.8% (SayCan) and 37.5% (Code as Policy).
• L3 Sub-task success: 80.5% manipulation, 80.6% navigation vs. best baselines of 60.9% and 57.9%.
• Constraint satisfaction (long-horizon, 40k+ tokens): OmniAct Full achieves 86.67% / 66.67% / 80.0% across Tool Use / Home Assistant / Manipulation scenarios; best ablation (Episodic Only) reaches 56.67% / 46.67% / 66.67%.
• Sliding Window compression ratio 16.3% but collapses to near-zero constraint satisfaction; OmniAct achieves 24.5% compression with substantially higher satisfaction.
• Token scalability: OmniAct stays within 2.5k–10k per-call tokens over 270 rounds; raw history exceeds 160k by round 260.
• Cross-model: Qwen3-VL-30B-A3B improves from 50.0% → 80.0% (+30.0 pts), reaching parity with vanilla Gemini-3.1-Pro (76.7%); Gemini-3.1-Pro itself improves +3.3 pts; Qwen3-VL-8B shows only marginal gain (+6.7), indicating a backbone capability floor.

Limitations

• Evaluated on a self-constructed 40-task benchmark with human-adjudicated success; no standardized embodied benchmark (e.g., BEHAVIOR-1K, ScanQA) used.
• Only two robotic platforms tested; generalization to dexterous hands, legged robots, or manipulation with force/tactile feedback is unvalidated.
• Qwen3-VL-8B (~8B params) marks a capability floor below which structured JSON tool dispatch degrades, but the threshold is not precisely characterized.
• Asynchronous visual preemption sampling frequency is not ablated; its interaction with high-speed manipulation or time-critical tasks is unaddressed.
• RoboBrain2 baseline underperformance is attributed partly to absent IoT training data — a confounder not fully disentangled from architectural differences.

Relevance to Agentic AI / LLM Agents

OmniAct directly addresses core challenges in long-horizon LLM agents: context management at scale and closed-loop verification of tool/action outcomes. Its event-boundary-driven memory compression is a concrete mechanism for preventing the token-blow-up that plagues multi-step agentic pipelines, and the reflective memory layer instantiates a lightweight continual learning loop without fine-tuning — both patterns transferable beyond robotics to software agents. The finding that mid-scale open-weight models match proprietary performance under structured orchestration strengthens the case that architectural discipline (separation of planning, memory, verification) is a higher-leverage intervention than raw model scale. The asynchronous preemption engine also models a general pattern for agents that must detect and recover from tool execution failures in the real world.