Skip to content

GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GauDP introduces a Gaussian-image synergistic representation for embodied multi-agent manipulation, reconstructing a shared 3D Gaussian field from each agent's RGB views and redistributing task-relevant Gaussian features back to individual agents. This enables pixel-aligned fusion of local and global context within a diffusion policy, achieving strong multi-arm coordination using only RGB input without requiring point clouds or camera poses at inference.

Problem

Existing multi-agent manipulation policies must choose between local observations (precise but uncoordinated) or global observations (consistent but low-resolution). Naive fusion of both lacks 3D structural constraints and cannot scale to varying agent counts. Point-cloud-based methods provide 3D structure but require dedicated 3D sensors. No prior image-based approach successfully integrates globally consistent scene geometry with per-agent local precision.

Method

GauDP operates in two stages. First, a fine-tuned Noposplat feed-forward network takes synchronized multi-view RGB images from all agents and predicts a set of 3D Gaussian primitives (position, scale, rotation, opacity, color, spherical harmonics) for each pixel using a shared ViT encoder + cross-view ViT decoder. Fine-tuning adds depth-supervised reconstruction loss L_rec = L_rgb + α·L_depth. Second, each agent's Gaussians—those derived from its own view but already enriched with cross-view context via cross-attention—are selected and transformed back into a 2D spatial grid matching the local image dimensions. These global context features are concatenated with local image features and fused via a lightweight convolutional module, then fed as conditioning into a standard DDPM Diffusion Policy for action prediction. At inference, only RGB inputs are needed; depth maps and camera poses are used only during Noposplat fine-tuning.

Key Contributions

  • Gaussian-image synergy representation that builds a globally consistent 3D scene from decentralized RGB views and selectively dispatches per-agent Gaussian subsets.
  • Pose-free, sparse-view 3D reconstruction at inference via fine-tuned Noposplat, extended with depth supervision for robotic manipulation domains.
  • Pixel-level (rather than coarse feature-level) fusion of local image features and redistributed Gaussian features within the diffusion policy backbone.
  • Architecture that scales to variable agent counts without structural changes, demonstrated on 2-, 3-, and 4-arm configurations.
  • Real-robot validation across three multi-agent tasks (Card Box Stacking, Card Box Handover, Grab Roller).

Results

  • Overall average success rate: GauDP 19.67% vs. best image-based baseline DP 5.00% and best point-cloud baseline DP3(XYZ+RGB) 14.33% on RoboFactory (6 tasks, 2–4 arms).
  • Lift Barrier (2 arms): GauDP 72% vs. DP 9%, DP3 30%.
  • Align Camera (2 arms): GauDP 26% vs. DP 3%, DP3(XYZ+RGB) 18%.
  • Place Food (2 arms): GauDP 15% vs. DP 12%, DP3 21%.
  • Reconstruction quality (fine-tuned vs. pretrained Noposplat): PSNR 23.42 vs. 17.92, SSIM 0.779 vs. 0.580, LPIPS 0.148 vs. 0.492.
  • Real-robot: GauDP 17/30 vs. DP 11/30 on Card Box Stacking; 19/30 vs. 14/30 on Card Box Handover; 27/30 vs. 22/30 on Grab Roller.
  • Inference speed: GauDP 1.28 FPS vs. DP 1.49 FPS, DP3 1.57 FPS (minor overhead); training time 6.5 GPU-h vs. DP 4.8, DP3 2.5.

Limitations

  • Inference speed is slightly reduced (1.28 vs. ~1.5 FPS) due to Gaussian reconstruction overhead.
  • Training time is longer (6.5 GPU-h vs. 2.5–4.8 GPU-h for baselines).
  • Tasks with 3–4 arms remain low overall (all methods near 0–9%), indicating coordination difficulty is not fully solved.
  • Stack Cube tasks remain near 0% for all methods including GauDP, suggesting the approach does not solve precision-critical stacking.
  • Depth maps and camera poses are required during fine-tuning, tying the method to simulation environments where these are available.
  • No evaluation on open-vocabulary or language-conditioned tasks; integration with VLA models is listed as future work.

Relevance to Agentic AI / LLM Agents

GauDP directly addresses the multi-agent perception-action bottleneck in embodied settings: how individual agents maintain both local precision and global situational awareness without centralized sensing. The selective Gaussian dispatch mechanism is a concrete instantiation of the broader principle of structured information sharing among agents—each agent receives a task-relevant subset of the shared world model rather than a full broadcast. For researchers tracking agentic AI, this work is a strong example of coupling a world-model component (3D Gaussian scene) with policy learning in a decentralized multi-agent loop, and the authors explicitly flag integration with VLA models as a next step, directly connecting to language-conditioned multi-agent embodied agents.