Skip to content

Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy

🕒 Published (v1): 2026-06-25 16:36 UTC · Source: Arxiv · link

Why this paper was selected

Meta-harness orchestrating heterogeneous cyber+physical tools with autonomous recovery; core scaffolding architecture

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OmniAct is a hierarchical embodied-agent framework that unifies cyber (API/IoT) and physical (manipulation/navigation) action spaces under a single orchestration layer, with event-boundary-driven memory compression and asynchronous visual failure detection. It achieves near-flat token consumption over 270+ interaction rounds while outperforming both monolithic VLA and LLM-based planning baselines on 40 real-world long-horizon tasks. Mid-scale open-weight models reach proprietary-level performance when wrapped in OmniAct's architecture, without additional training.

Problem

Existing embodied systems have three simultaneous failure modes: (1) VLM planners treat cyber-tool invocation and physical manipulation as separate pipelines with brittle hand-crafted interfaces; (2) agent frameworks accumulate interaction history linearly, causing context overflow and semantic conflicts between long-term preferences and recent state; (3) VLA policies are open-loop—undetected physical failures cascade into irreversible downstream faults.

Method

OmniAct decomposes the agent into three cooperating modules:

Multimodal Semantic Planner — Fuses speech, vision, and discrete sensor state into a multimodal context \(c_t = (v_t, s_t)\) and samples a skill-routing action \(a_t = (\tau_t, g_t) \sim \pi(\cdot \mid c_t, H_t, \mathcal{T})\), where \(\tau_t\) selects a skill from inventory \(\mathcal{T}\) and \(g_t\) are conditioning parameters. Output is a structured JSON spec supporting parallel multi-skill dispatch in a single step.

Asynchronous Visual Preemption Engine — A low-frequency VLM monitor runs concurrently with physical execution, periodically sampling frames and querying for semantic anomalies (e.g., "is the object still grasped?"). On anomaly detection, it halts the VLA executor and returns control to the planner with failure context, closing the semantic loop without force sensing.

Adaptive Hierarchical Memory — Three tiers: (a) Sensory: FIFO frame buffer for the control policy; (b) Episodic: event-boundary-driven compression—when a semantic boundary is detected (sub-task completion, state change), raw turns are compressed into timestamped keyframe summaries, yielding sub-linear context growth proportional to distinct semantic events rather than raw turns; (c) Reflective: stores failure root-cause analyses and corrective strategies; relevant reflections are retrieved and injected at planning time, converting isolated recoveries into cumulative adaptation.

Key Contributions

  • Unified cyber-physical skill-routing action space that dispatches heterogeneous executors (IoT, web APIs, VLA policies, navigation) from a single planner without domain-specific bridges
  • Event-boundary-driven episodic compression maintaining per-call token usage in a 4k–8k band over 270 rounds vs. linear growth to 160k+ tokens for raw history
  • Asynchronous visual preemption providing semantic-level closed-loop verification for continuous physical execution without force/proprioceptive sensing
  • Reflective memory layer that transforms per-episode failure recovery into persistent, retrievable corrective knowledge
  • Real-world validation on two distinct morphologies (UR5e + Keenon mobile robot) with four IoT devices across 40 tasks at three complexity levels

Results

  • L3 (Full Multimodal) E2E success: 50.0% (manipulation) and 54.2% (navigation) vs. best baselines 43.8% (SayCan, manipulation) and 37.5% (Code as Policy, navigation)
  • L3 Sub-task success: 80.5% (manipulation) and 80.6% (navigation) vs. best baselines 60.9% and 57.9%
  • Memory scalability: per-call input tokens plateau at 2.5k–10k across 270 rounds; raw history exceeds 160k tokens by round 260
  • Constraint satisfaction with full memory: 86.67% (tool use), 66.67% (home assistant), 80.0% (manipulation) at 24.5% context compression ratio vs. 66.67%/46.67%/66.67% for raw history
  • Cross-model lift: Qwen3-VL-30B-A3B improves from 50.0% → 80.0% (+30.0 pp), reaching parity with vanilla Gemini-3.1-Pro; Gemini-3.1-Pro still gains +3.3 pp
  • RoboBrain2 (32B, SOTA on established benchmarks) substantially underperforms OmniAct on cross-modal tool coordination, validating that orchestration is a separate competency from spatial reasoning

Limitations

  • Requires backbone models capable of producing well-formed structured JSON outputs; Qwen3-VL-8B shows only +6.7 pp gain, establishing an approximate lower bound on backbone capability
  • No force/tactile sensing—failure detection relies entirely on visual preemption, which may miss failures invisible to cameras (e.g., slip under occlusion)
  • Benchmark is 40 tasks (3 trials each), human-adjudicated; scale and inter-rater reliability are limited
  • IoT domain training data absence degrades generalization for models like RoboBrain2, suggesting the task suite may be narrow relative to deployment diversity
  • Event-boundary detection is not formally specified; its robustness to ambiguous or overlapping sub-task boundaries is not evaluated

Relevance to Harnesses / Meta-Harnesses

OmniAct is directly a meta-harness architecture: it wraps heterogeneous skill executors (VLA policies, API clients, IoT drivers) under a unified orchestration layer that handles routing, memory, and verification without modifying the sub-skills. The three-tier memory design—sensory buffer, episodic compression, reflective store—is a concrete, empirically validated instantiation of the pattern where a meta-harness accumulates and compresses execution history to stay within context bounds across arbitrarily long runs. The asynchronous visual preemption engine is a harness-level verification module that monitors sub-skill execution and triggers replanning, directly analogous to the evaluator/verifier stage in agent harnesses. The result that architectural orchestration (not model scaling) closes the open-weight/proprietary gap is a strong empirical argument for the meta-harness paradigm itself.