IDEA: Insensitive to Dynamics Mismatch via Effect Alignment for Sim-to-Real Transfer in Multi-Agent Control¶

🕒 Published (v1): 2026-06-25 03:49 UTC · Source: Arxiv · link

Why this paper was selected

Effect-alignment for sim-to-real in multi-agent control; addresses dynamics mismatch without policy retraining

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

IDEA addresses the sim-to-real gap in multi-agent reinforcement learning by lifting policy execution to a semantic abstraction level: discrete semantic actions are mapped to continuous control via onboard closed-loop controllers, bounding the sim-to-real performance gap without modeling low-level dynamics. An inter-agent action synchronization mechanism eliminates temporal drift during decentralized deployment. Zero-shot transfer experiments on four navigation tasks with real UGVs show >20% higher success rates and 0% collision rate over domain randomization and domain adaptation baselines.

Problem¶

Standard MARL policies trained in simulation operate in low-level continuous control spaces that are highly sensitive to dynamics mismatch (nonlinear friction, hysteresis, sensor latency). Domain randomization (DR) is bounded by what parameters can be perturbed in the simulator and hurts convergence speed; online adaptation methods (RMA, HIM) require recurrent history encoders that introduce inference delay and compounding errors. Additionally, decentralized real-world execution causes inter-agent action timing drift that violates the discrete-time Dec-POMDP assumption underlying MARL training.

Method¶

IDEA instantiates a context-conditioned Dec-POMDP $\mathcal{M} = \langle \mathcal{N}, S, A, P, R, O, C, \gamma \rangle$ where context $c \in C$ encodes static geometric configurations sampled at episode start.

Effect Alignment. The policy $\pi(a \mid o)$ outputs discrete semantic actions (forward/back/turn-left/turn-right/stop). A decentralized low-level closed-loop controller $k$ maps each semantic action to a continuous command $u_t = k(o_t, a_t)$, inducing a closed-loop transition kernel: $$\tilde{P}(s_{t+1} \mid s_t, a_t, c) = \mathbb{E}_{o_t \sim \mathcal{O}}\left[P(s_{t+1} \mid s_t, u_t, c)\right]$$ If closed-loop control bounds the expected state metric distance between simulation and real next states as $\mathbb{E}\|s^{t+1}_{\text{sim}} - s^{t+1}_{\text{real}}\|_1 \leq \epsilon$, then by the $W_1$ Wasserstein bound and the Simulation Lemma, the value-function performance gap satisfies: $$|V^\pi_{\text{real}}(c) - V^\pi_{\text{sim}}(c)| \leq \frac{\gamma L_V}{1 - \gamma} \epsilon$$

Action Synchronization. A designated coordination node collects readiness signals $m^i_t$ from all agents before broadcasting a synchronized execution trigger, ensuring simultaneous joint action execution.

Parallel Training. Isaac Gym is extended with episode-level static context assignment across $N$ parallel environments; geometric configurations are never resampled mid-episode, preserving Dec-POMDP validity while maximizing layout diversity across the training distribution $\mathcal{D}_{\text{train}}$.

Key Contributions¶

Theoretical sim-to-real performance bound via $W_1$ distance + Simulation Lemma, showing that closed-loop semantic action alignment bounds the value-gap without dynamics modeling.
A communication-based action synchronization protocol that achieves zero inter-agent step difference during real deployment.
A GPU-tensor-based parallel Isaac Gym training platform supporting large-batch MARL with diverse geometric contexts under limited compute.

Results¶

Zero-shot transfer success rate: IDEA achieves 100% (Random Goal), 90% (Cross Obstacles), 90% (Narrow Gate), 80% (Maze Game) vs. DR: 30/70/40/0%, RMA: 50/70/40/50%, HIM: 50/50/40/60%.
Action effect fidelity: IDEA's translational action mismatch <5% (e.g., forward full step: 0.194 ± 0.002 m vs. target 0.20 m); baselines deviate 30–35% (DR: 0.134 ± 0.013 m).
Inter-agent timing: IDEA achieves 0 step difference across all tasks; baselines show up to 11.5 ± 11.03 steps discrepancy (HIM, Maze Game).
Collision rate: IDEA = 0% across all tasks; baselines reach up to 50% (DR, Maze Game).
Timeout rate: IDEA ≤ 20%; DR reaches 50–60% on several tasks.
Training convergence: IDEA converges faster than DR, RMA, and HIM in simulation learning curves (all four tasks).

Limitations¶

Policy operates over a predefined, fixed discrete semantic action space; finer-grained or continuous semantic control is deferred.
Geometric context is static within each episode; dynamic obstacle tracking is not supported.
Action synchronization introduces a temporal overhead that increases total task execution time.
Experiments use only a two-UGV system; scalability to larger swarms is untested.
Low-level controller design (PID or equivalent) is not ablated; effect-alignment quality depends on controller fidelity.

Relevance to Agentic AI / LLM Agents¶

IDEA's core idea — abstracting execution to a discrete semantic action space so that high-level policies remain robust to low-level physical variation — is directly analogous to how LLM-based agents issue tool calls or API actions whose low-level execution details are hidden behind abstractions. The theoretical treatment (bounding value-gap via $W_1$ distance under semantic action alignment) offers a formal lens for thinking about robustness of agentic systems to environmental distribution shift. The action synchronization mechanism tackles a form of multi-agent coordination challenge — ensuring temporal consistency of joint actions during decentralized execution — that appears in any distributed agentic system where multiple agents must coordinate on a shared timeline. This work is most relevant to researchers studying grounded, embodied multi-agent systems where policies must transfer zero-shot to novel deployment conditions.