Skip to content

CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CAML extends Auxiliary Modality Learning (AML) to multi-agent settings by enabling \(N\) agents to share full multi-modal data during training while each agent performs inference with a reduced modality subset (\(|I_{test}| \leq |I_{train}|\)) at test time. A teacher model trained on all agents' combined modalities distills knowledge into a student model via knowledge distillation (KD), allowing robust reduced-modality deployment even when agent count differs between train and test.

Problem

Single-agent AML frameworks cannot exploit multi-agent complementarity: in CAV scenarios, individual agents suffer occlusions and limited sensor range, creating decision-making blind spots. Conversely, existing multi-agent collaboration work assumes consistent modality access at both train and test time, making it fragile to sensor failures or resource-constrained deployments. No prior framework unifies missing-modality robustness with multi-agent collaboration.

Method

CAML uses a teacher–student KD pipeline over a multi-agent graph. During training, a teacher model gives each agent \(A_i \in A_{train}\) access to all \(K\) modalities; agents independently encode their observations, share embeddings (centralized: all agents relay to one ego; decentralized: \(k\)-nearest neighbors), aggregate per-modality embeddings, fuse cross-modality representations via concatenation or cross-attention, and predict via a task head. A student model mirrors this architecture but each agent processes only \(L \leq K\) modalities (which can differ per agent). KD transfers the teacher's richer joint-agent, full-modality representation into the student by minimizing a distillation loss. At test time only the student runs, using whatever modalities are available (\(I_{test} \subset I_{train}\)), with a potentially different number of agents (\(M \neq N\)). An information-theoretic analysis in the appendix shows joint multi-agent observations increase mutual information with the ground truth relative to single-agent observations.

Key Contributions

  • First framework to jointly address missing-modality inference and multi-agent collaboration; unifies AML and V2V/multi-robot communication paradigms.
  • Cross-agent, cross-modal KD: teacher encodes collaborative full-modality signal; student learns to reproduce it from partial, heterogeneous agent observations.
  • Flexible agent-count decoupling: training and test sets can have different numbers of agents (\(N \neq M\)), enabling real-world deployment where connected infrastructure is absent.
  • Demonstrated on two distinct task types: collaborative decision-making (CAV braking) and collaborative semantic segmentation (aerial-ground robots).
  • Lower communication overhead than prior collaborative baselines: 1.0 KB shared package size vs. 4.9 KB (STGN) and 65.5 KB (COOPERNAUT); 3.7 ms inference latency vs. 18.5 ms and 90.0 ms.

Results

  • CAV accident detection (ADR), vs. single-agent AML baseline: +13.0% (overtaking), +34.6% (left turn), +58.1% (red light violation).
  • CAV ADR, vs. COOPERNAUT (multi-agent, LiDAR train+test): up to +21.5%.
  • CAV ADR, vs. STGN (multi-agent, RGB+Depth train+test): +10.9% in left-turn scenario despite CAML using only RGB at test time.
  • Generalization to single-agent test (trained multi-agent): CAML outperforms COOPERNAUT and STGN on both ADR and Expert Imitation Rate (EIR) across all three scenarios.
  • Collaborative semantic segmentation (aerial-ground robots): up to +10.6% mIoU.

Limitations

  • Decentralized system complexity scales as \(O(N^2D)\) in the worst case (all-to-all communication), which may be prohibitive for large fleets.
  • Training data for CAV experiments is generated from AUTOCASTSIM, a simulation benchmark; real-world transfer is unvalidated.
  • Assumes modality availability pattern is known at training time so the student can be configured appropriately; arbitrary runtime modality drop is not addressed.
  • The paper is truncated before ablation results are fully reported, so the marginal contribution of each design choice (KD loss weight, attention type, centralized vs. decentralized) is not fully visible.

Relevance to Harnesses / Meta-Harnesses

CAML is essentially a training harness that orchestrates a heterogeneous fleet of agents with different modality access patterns, coordinating shared representation learning via a structured teacher–student pipeline — a form of meta-harness for multi-agent perception. Its teacher model acts as the harness's privileged coordinator, distilling a globally coherent signal into individually constrained agents, analogous to how a meta-harness manages subagent outputs under resource constraints. The explicit decoupling of train-time topology (agent count, modality set) from test-time topology is directly relevant to harness designers who must plan for degraded-environment deployment. The centralized vs. decentralized communication modes map cleanly onto orchestrator patterns (single dispatcher vs. peer-to-peer mesh) seen in agentic harness architectures.