Pragmatic Heterogeneous Collaborative Perception via Generative Communication Mechanism

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GenComm introduces a generation-based communication mechanism for heterogeneous multi-agent collaborative perception where each ego agent locally generates feature representations for collaborators using received spatial messages via a conditional diffusion model, without modifying any agent's pretrained network. This non-intrusive design achieves state-of-the-art detection performance on OPV2V-H, DAIR-V2X, and V2X-Real while reducing the parameter and FLOPs cost of integrating a new agent by 81% and 62% over leading adaptation- and reconstruction-based methods, respectively.

Problem

In real-world autonomous driving, collaborative perception agents have heterogeneous sensors and model architectures, creating domain gaps when sharing intermediate BEV features. Existing methods fail on two axes: (1) adaptation-based approaches (MPDA, HEAL, STAMP) require retraining encoders or fusion networks, disrupting established semantic consistency; (2) both adaptation and reconstruction-based methods (CodeFilling) incur high incremental computational/parameter costs when new agents join, limiting scalability. No prior method simultaneously preserves semantic consistency, avoids intrusive retraining, and supports low-cost plug-in of new agents.

Method

GenComm operates in two training stages:

Stage 1 (homogeneous pre-training): Three modules are jointly trained end-to-end within a homogeneous collaborative setting: - Deformable Message Extractor (DME): Uses deformable convolutions with learned offset fields to extract compressed spatial confidence maps from BEV features, which are transmitted in place of full intermediate features. A learnable resizer adapts messages to the receiver's resolution. - Spatial-Aware Feature Generator (SAFG): A conditional diffusion model (conditional U-Net) at the ego agent that takes the received spatial messages as conditioning and generates features aligned to the ego's semantic space via iterative denoising over T steps (MSE loss against ground-truth ego features). - Channel Enhancer (CE): Refines generated features channel-wise using PConv, a gating mechanism splitting features into modifiable/static parts, and channel attention (global average pooling + FC layers) to suppress redundant channels.

Stage 2 (heterogeneous alignment): Only a lightweight per-collaborator DME is fine-tuned using classification and regression losses to correct numerical discrepancies (e.g., inconsistent confidence scores across heterogeneous BEV maps). The core generator and fusion network remain frozen.

At inference, collaborators transmit only spatial messages (not full feature maps), and the ego agent generates collaborator features locally before fusion.

Key Contributions

• First generation-based communication mechanism for heterogeneous collaborative perception; no modification to any agent's pretrained encoder or fusion network.
• Deformable Message Extractor that captures modality-agnostic spatial information as a compressed, transmittable representation.
• Conditional diffusion-based Spatial-Aware Feature Generator that produces ego-semantic-space-aligned features from spatial messages.
• Two-stage training strategy separating homogeneous generative pre-training from lightweight heterogeneous numerical alignment.
• 81% reduction in parameters and FLOPs versus STAMP (best adaptation baseline) when integrating new agents; 62% reduction versus CodeFilling (reconstruction baseline).
• Reduced communication volume (log₂ = 16 vs. 22 for most baselines) alongside improved detection accuracy.

Results

• OPV2V-H (AttFuse backbone, L⁶⁴_P–L³²_S setting): GenComm AP50/AP70 = 0.8043/0.6332 vs. best baseline BackAlign 0.7873/0.5841.
• OPV2V-H (V2X-ViT backbone, L⁶⁴_P–L⁴⁰_S setting): GenComm AP50/AP70 = 0.7630/0.5759 vs. best baseline STAMP 0.7508/0.5442.
• DAIR-V2X (V2X-ViT, L⁶⁴_P–CE): GenComm AP30/AP50 = 0.5651/0.4665 vs. STAMP 0.5421/0.4935 (STAMP slightly higher at AP50 here).
• V2X-Real scalability (4-agent, L¹²⁸_H+L¹²⁸_L+L¹²⁸_M+L¹²⁸_T): GenComm AP30/AP50 = 0.7144/0.6362 vs. best baseline BackAlign 0.6352/0.5896.
• Scalability (Table 2): GenComm #Params = 0.31M, #FLOPs = 0.615G per new agent vs. STAMP 1.64M/3.084G and MPDA 5.75M/51.93G.
• Robustness: Superior AP under Gaussian pose noise and time delay (0–500ms) compared to all baselines on OPV2V-H.

Limitations

• Requires bilateral consensus between vendors before deploying heterogeneous DMEs; commercial competition or adversarial incentives may prevent this in practice.
• The non-fully-connected communication graph assumption (more realistic) is addressed in the application rationale but not fully validated in controlled experiments.
• Diffusion-based generation introduces inference latency; ablation shows channel size vs. inference time trade-off but overall latency impact at deployment scale is not deeply analyzed.
• Evaluation is restricted to LiDAR and camera modalities on three datasets; generalization to other sensor types (radar, thermal) is not demonstrated.

Relevance to Agentic AI / LLM Agents

GenComm addresses a fundamental multi-agent systems challenge: enabling heterogeneous agents with different internal representations to collaborate effectively without centralizing control or requiring agents to expose or modify their private models. The plug-and-play, non-intrusive design pattern—where only a lightweight per-peer adapter (DME) is added and the core model remains frozen—is directly analogous to adapter-based specialization strategies in LLM agent frameworks. The generative communication paradigm, where an agent synthesizes a peer's representation locally from compressed spatial signals, parallels how LLM agents might use local world models to simulate collaborator states rather than sharing raw activations. The two-stage training (homogeneous pre-training → heterogeneous lightweight alignment) mirrors the pretrain-then-finetune paradigm central to modern agentic pipelines, making this work a strong reference for scalable, privacy-preserving multi-agent coordination.