Skip to content

VULCAN: Tool-Augmented Multi Agents for Iterative 3D Object Arrangement

🕒 Published (v1): 2025-12-26 19:22 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VULCAN is an agentic pipeline for iterative 3D object arrangement that combines MCP-based visual tool APIs, a constraint-based collision-free pose solver, and a three-role multi-agent framework (Planner, Executor, Evaluator). It decomposes complex, multi-step arrangement instructions into sequential actions with adaptive backtracking to recover from dead ends. On a 25-scene, 111-task benchmark, it eliminates collisions and floating artifacts entirely while outperforming all baselines on plausibility and instruction consistency.

Problem

Prior MLLM-based 3D arrangement methods are limited to single-step operations: one comprehensive edit per instruction. This fails on tasks requiring sequential object movement where intermediate states must remain physically valid and serve as preconditions for subsequent steps. Raw code-generation or simplified scene description interfaces burden the MLLM with either unstructured 3D data or insufficient spatial information, causing error propagation across steps.

Method

VULCAN structures each arrangement step as a Plan–Execute–Evaluate loop over three specialized agents:

  • Planner: Receives global context (user instruction + rendered image history of all prior steps) and outputs a natural-language sub-instruction plus a normalized 2D pixel coordinate for the next object placement.
  • Executor: Translates the Planner's 2D coordinate into a physically valid 3D pose via three MCP-exposed visual probing tools (ListObjectsInArea, RayProbe, RenderWithHighlight), then assembles a geometric constraint list (CloseToPix, Contact, NoOverhang, Distance, FaceTo, Rotate) and runs an AdamW-based optimization solver. The solver samples perturbed target coordinates, optimizes constraint losses, filters by error threshold Ď„, then applies a collision detector, returning the lowest-error collision-free pose.
  • Evaluator: Three independent Evaluator agents score the resulting render on a five-point scale (terrible→excellent); consensus filtering rejects solutions with non-positive average score. A deterministic floating check supplements MLLM judgment.

Adaptive backtracking maintains an anchor step that resets to half the current depth on failure and advances when a new maximum sequence length is achieved, pruning the exponential search space without exhaustive enumeration.

Key Contributions

  • MCP-based visual API toolset for interactive 3D scene probing, integrated via Blender-MCP, replacing brittle raw-code manipulation with function-level calls.
  • Constraint-based collision-free pose solver using AdamW optimization over a geometric constraint vocabulary, with preconditioning via perturbed pixel sampling.
  • Three-role collaborative multi-agent framework separating global planning, per-step execution, and intermediate evaluation; Planner holds global context while Executor/Evaluator operate locally.
  • Adaptive backtracking search algorithm with a dynamically adjusted anchor step for recovery from dead ends.
  • A new benchmark of 25 scenes / 111 unit tasks from BlenderKit, InfiniGen, and BlenderGym, explicitly designed for multi-step interdependent arrangement evaluation.

Results

  • Collision rate: Ours 0.000 vs. Blender-MCP 0.459, BlenderAlchemy 0.631, FirePlace 0.513.
  • Floating rate: Ours 0.000 vs. Blender-MCP 0.774, BlenderAlchemy 0.676, FirePlace 0.225.
  • Plausibility (0–4): Ours 3.796 vs. FirePlace 3.515, BlenderAlchemy 3.368, Blender-MCP 3.348.
  • Consistency (0–4): Ours 3.592 vs. FirePlace 3.135, Blender-MCP 2.973, BlenderAlchemy 2.770.
  • Human study (30 subjects): VULCAN win rate 62.1%/65.9%/70.0% vs. BlenderAlchemy, 60.5%/59.0%/62.9% vs. Blender-MCP, 58.8%/54.4%/58.0% vs. FirePlace across Consistency/Plausibility/Physical plausibility.
  • Ablation: Removing the multi-tool library raises collision to 0.495; removing backtracking produces non-zero collision (0.036) and floating (0.054); removing MCP tools entirely raises collision to 0.603; single-agent variant achieves zero collision/floating but drops plausibility to 3.623 and consistency to 3.328.

Limitations

  • Single-camera view only; tasks requiring novel-view observations (e.g., placing objects in occluded spaces behind walls) are unsupported.
  • Benchmark is small (25 scenes, 111 tasks), all in Blender; generalization to real-world 3D reconstruction pipelines is unvalidated.
  • Evaluator agents exhibit hallucinated ratings requiring consensus filtering as a workaround, indicating unreliable per-agent judgment.
  • Backtracking search complexity is not formally analyzed; worst-case behavior on deeply interdependent tasks is uncharacterized.
  • Dependent on Gemini-2.5-pro specifically; robustness across other MLLMs is not reported.

Relevance to Harnesses / Meta-Harnesses

VULCAN is a concrete instance of a domain-specific agentic harness: it orchestrates multiple MLLM sub-agents with fixed roles (Planner, Executor, Evaluator), wraps external tools behind a standardized MCP API layer, and implements a control loop with explicit retry/backtracking logic—all architectural patterns central to harness design. The use of MCP as the tool-integration protocol directly instantiates the "harness as tool-broker" pattern, where the harness mediates between agent reasoning and external capabilities (3D physics solver, rendering engine) rather than exposing raw APIs. The consensus-based Evaluator and deterministic floating check illustrate a hybrid verification pattern (learned + rule-based) that meta-harnesses can adopt for quality gating without full reliance on any single judgment source.