Skip to content

EMOS: Embodiment-aware Heterogeneous Multi-robot Operating System with LLM Agents

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EMOS is an LLM-based multi-agent framework for heterogeneous multi-robot systems (HMRS) that achieves embodiment-aware task planning by having each robot's agent self-generate a structured "Robot Resume" from its URDF file, rather than relying on human-assigned roles. The framework operates in a three-stage pipeline: scene context construction, centralized group discussion, and decentralized parallel action execution. The accompanying Habitat-MAS benchmark provides the first simulated evaluation environment for embodiment-aware heterogeneous multi-robot collaboration.

Problem

Existing LLM-based multi-agent systems assign agents to fixed human-designed roles, which cannot capture the physical capability constraints of heterogeneous robots (e.g., a wheeled robot cannot climb stairs; a drone lacks a gripper). This conflation of role with physical embodiment prevents "Level 4" fully-automated HMRS, where the system autonomously handles task decomposition, coalition formation, task allocation, and execution without human-crafted protocols. No prior simulated benchmark existed to measure embodiment-aware reasoning across diverse robot types.

Method

Robot Resume generation: Each robot's URDF is preprocessed into a skeleton tree (stripping visual/collision/inertial tags to reduce context length), then an LLM agent reads the skeleton to produce a natural-language capability summary. In parallel, the agent calls a forward kinematics API to compute numerical capability descriptors — for example, the arm workspace is represented as a convex hull of sampled end-effector positions in 3D space, yielding a center/radius representation. The resume encodes three capability axes: mobility, perception, and manipulation, each as a JSON object with a prose summary and numeric fields.

Hierarchical execution pipeline (Algorithm 1): 1. Centralized Group Discussion (synchronous): A central LLM planner ingests all robot resumes, the scene graph (textual region-connectivity graph \(G=(V,E)\)), and task \(T\), then assigns subtask \(\text{subtask}_i\) to each robot \(r_i\). Each robot's dedicated agent reflects on subtask feasibility given its resume; infeasible assignments trigger replanning. 2. Decentralized Action Execution (asynchronous): Each agent independently runs a function-calling loop — \(\text{action}_i \leftarrow \text{FunctionCall}(r_i, \text{subtask}_i, \text{history}_i)\) — against robot-specific skill libraries, appending (action, response) pairs to its history until TaskFinished.

Scene context is a four-layer representation (region connectivity graph L1, semantic mesh L2, agent/object states L3, navmesh L4) converted to text; L2 and L4 remain geometric and drive low-level trajectory planning.

Key Contributions

  • EMOS framework: First LLM-based MAS for HMRS that uses self-prompted, bottom-up robot capability generation (Robot Resume) instead of human-assigned role play.
  • Robot Resume generation pipeline: Hybrid LLM summarization + forward kinematics tool-calling to produce structured JSON capability descriptors directly from URDF.
  • Habitat-MAS benchmark: First simulated benchmark for embodiment-aware heterogeneous multi-robot collaboration; covers 61 scenes (27 Matterport3D + 34 HSSD), four robot types (Fetch, Stretch, Drone, Spot), and four task categories (manipulation, perception, navigation, multi-floor rearrangement) with episodes gated on physical capability requirements.
  • Ablation evidence: Experimental results demonstrate that both the robot resume and the hierarchical (centralized planning + decentralized execution) design are individually necessary for HMRS performance.

Results

  • The paper reports experimental results on Habitat-MAS demonstrating that removing robot resumes causes performance degradation across all four task types, and that the hierarchical design is essential.
  • Specific numeric results (task success rates, comparison tables against baselines) are in sections not included in the provided excerpt; the text states results "indicate that the robot's resume and the hierarchical design... are essential."
  • No external baseline systems are named in the provided text; the benchmark itself is the primary evaluation vehicle.

(Note: headline accuracy numbers are not present in the provided excerpt.)

Limitations

  • Perfect SLAM assumption: Stage 1 scene context construction assumes an ideal semantic SLAM system; the authors acknowledge this explicitly and note real deployment would rely on Hydra-multi.
  • Simulation gap: Habitat-MAS is fully simulated; transfer to real heterogeneous hardware remains unvalidated.
  • Text-only environment interface: The benchmark provides textual environment descriptions rather than raw sensor inputs, sidestepping low-level perception challenges.
  • No fine-tuning or training: The system relies entirely on pretrained LLM zero-shot/few-shot capabilities; performance is bounded by the base model's spatial and kinematic reasoning.
  • Fixed robot set: Only four robot types are evaluated; the resume pipeline's generalization to arbitrary URDF structures is not stress-tested at scale.

Relevance to Harnesses / Meta-Harnesses

EMOS is a direct instance of a meta-harness: it orchestrates multiple sub-agents (one per robot), each with its own skill library and execution loop, under a centralized planner that performs task decomposition and assignment — exactly the harness-of-harnesses pattern. The Robot Resume generation pipeline is itself a tool-augmented self-configuration step, where an agent calls structured APIs (forward kinematics) to generate its own operational context before the harness runs, analogous to how meta-harnesses might auto-discover or auto-configure component capabilities. The three-stage pipeline (context construction → centralized planning → decentralized execution) mirrors the fetch→plan→dispatch→collect pattern common in software meta-harnesses, but grounded in physical robot constraints rather than software task roles. For researchers building harnesses that must adapt to heterogeneous backend capabilities, the Robot Resume approach — structured self-description via tool-calling, consumed by a coordinator — is a transferable design pattern.