Skip to content

CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CAML (Collaborative Auxiliary Modality Learning) is a multi-modal multi-agent framework that lets agents share full-modality data during training while supporting inference with reduced modalities at test time. It uses knowledge distillation from a full-modality teacher to a reduced-modality student, enabling robust deployment in sensor-constrained or failure scenarios. Evaluated on connected autonomous driving and aerial-ground robot segmentation, CAML yields up to 58.1% improvement in accident detection over single-agent AML baselines.

Problem

Prior work splits into two silos: (1) Auxiliary Modality Learning (AML) addresses train-time auxiliary / test-time reduced modalities but only for single agents, leaving blind spots due to limited sensor coverage; (2) multi-agent collaboration frameworks assume consistent modality access at both train and test time, which fails under sensor failures or resource constraints. No prior framework jointly addresses missing modalities and multi-agent collaboration.

Method

CAML trains a teacher model where all N agents share full multi-modal embeddings (e.g., RGB + LiDAR) via either centralized (ego aggregation) or decentralized (k-nearest neighbor) communication. A student model mirrors the teacher's architecture but each agent processes only a subset of modalities (e.g., RGB only), sharing those partial embeddings among agents to form a joint multi-modal representation. Knowledge distillation transfers the teacher's richer representation to the student via a KD loss alongside the task loss (binary cross-entropy for driving, cross-entropy for segmentation). At test time only the student runs, with reduced modalities and a potentially different number of agents than training. The number of agents and modalities can differ between train and test phases by design.

Key Contributions

  • First framework unifying multi-agent collaboration with auxiliary modality learning (AML), enabling train-with-all / infer-with-fewer across agents.
  • Cross-agent, cross-modal knowledge distillation that encodes collaborative full-modality knowledge into per-agent reduced-modality student models.
  • Flexible agent-count generalization: trained with N agents, deployable with as few as a single agent.
  • Demonstrated on two distinct tasks: CAV accident detection (simulation, AUTOCASTSIM) and aerial-ground robot semantic segmentation (real-world, CoPeD dataset).
  • Lower communication overhead and inference latency than baselines: 1.0 KB package size and 3.7 ms latency vs. 65.5 KB / 90.0 ms (COOPERNAUT) and 4.9 KB / 18.5 ms (STGN).

Results

  • CAV accident detection (ADR) vs. AML (single-agent baseline): +13.0% overtaking, +34.6% left turn, +58.1% red light violation.
  • CAV vs. COOPERNAUT (multi-agent, LiDAR at test): up to +21.5% mean ADR; CAML uses only RGB at test.
  • CAV vs. STGN (multi-agent, RGB+Depth at test): CAML exceeds STGN mean ADR by 10.9% in left-turn despite fewer test-time modalities.
  • Single-agent generalization (multi-agent train, single-agent test): CAML outperforms COOPERNAUT and STGN in all three scenarios on both ADR and EIR.
  • Semantic segmentation (mIoU) vs. AML: +8.9% indoor, +10.6% outdoor on CoPeD.
  • Ablation — adding state modality: further improves ADR ~1–2% across scenarios.
  • Intermediate vs. late cooperation: CAML (intermediate fusion) outperforms late-cooperation KD baseline across all scenarios.

Limitations

  • Evaluated only on simulated CAV (AUTOCASTSIM) and one small real-world aerial-ground dataset (CoPeD, 2 robots); scalability to larger fleets is not empirically validated.
  • Communication cost analysis is theoretical; real V2V latency and packet loss under dynamic conditions are not tested.
  • The framework assumes cooperative agents that willingly share embeddings; adversarial or selfish agents are not considered.
  • Number of agents can vary between train and test, but the paper does not test extreme mismatches (e.g., 10 training agents → 1 test agent at scale).
  • Modality subsets at test time are fixed per experiment; dynamic per-agent modality dropout at inference is not explored.

Relevance to Agentic AI / LLM Agents

CAML directly addresses a core challenge in multi-agent systems: how agents should coordinate their heterogeneous sensory or knowledge inputs during training to remain robust at deployment when resources are constrained. While the paper focuses on perception-action agents (vehicles, robots) rather than LLM-based agents, the architectural pattern — collaborative training with shared representations distilled into lightweight per-agent inference models — is transferable to LLM agent pipelines where some agents may lack access to certain tools, memory, or modalities at runtime. The decentralized variant and the variable-agent-count design are especially relevant to agent systems where team composition is dynamic. For researchers tracking LLM agents, CAML is most useful as a principled foundation for handling partial observability and resource heterogeneity across agent teams.