Skip to content

Transductive Visual Programming: Evolving Tool Libraries from Experience for Spatial Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Serena Yeung-Levy (Stanford); ICLR 2026; evolving tool libraries for 3D spatial reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Transductive Visual Programming (TVP) is a closed-loop visual programming framework that grows its tool library by abstracting reusable functions from accumulated problem-solving experience, rather than speculating tools from problem descriptions alone. Applied to 3D spatial reasoning on Omni3D-Bench, TVP achieves 33.3% overall accuracy, outperforming GPT-4o by 22% and the prior best visual programming system by 11%.

Problem

Visual programming systems for spatial reasoning either rely on fixed, predefined toolsets (VisProg, ViperGPT) or perform speculative inductive tool creation before any problem-solving experience is available (VADAR). The inductive approach produces an open-loop system where tools are built without grounding in actual solution patterns; empirically, 94.2% of VADAR programs still fall back to basic predefined tools, indicating induced tools are rarely reused. TVP addresses this closed-loop gap.

Method

TVP maintains two co-evolving structures: an Example Library \(\mathcal{E}\) of high-quality program solutions and a Tool Library \(\mathcal{T}\) of callable functions. The system alternates between two phases over \(T\) iterations on a dataset \(\mathcal{D} = \{(I_i, q_i)\}_{i=1}^N\):

Phase I — Experience Accumulation: For each query \(q_i\), TVP (1) retrieves up to \(k_\text{max}\) similar past solutions from \(\mathcal{E}\) via text-embedding similarity above threshold \(\tau_\text{sim}\); (2) samples \(m\) candidate programs conditioned on retrieved examples and current \(\mathcal{T}\); (3) executes candidates in Python; (4) judges candidates with a VLM on execution soundness, coding logic, tool usage, and visual evidence. Programs scoring above \(\tau_q\) enter \(\mathcal{E}\), replacing lower-quality prior solutions for the same question.

Phase II — Transductive Tool Abstraction: Every \(n_a\) questions, \(\mathcal{E}\) is clustered by embedding similarity. Clusters exceeding size threshold \(\tau_\text{cluster}\) and abstraction-potential score \(\tau_\text{potential}\) trigger a tool abstractor LLM, which creates a parameterized function (e.g., compute_3d_ratio, find_closest_obj) capturing shared logic. Each candidate tool undergoes two-stage validation: 100% execution success on rewritten cluster programs (Stage 1), followed by a VLM correctness judge on divergent results (Stage 2); tools must exceed \(\tau_\text{correct}\) to enter \(\mathcal{T}\). At every \(n_d\) questions, functionally similar tools are identified and merged into more general abstractions (e.g., compute_3d_ratio + compute_3d_group_size_ratio → compute_objects_size_ratio), with merged tools re-validated against all affected examples. Basic tools (GroundingDINO loc, UniDepth depth, GPT-4o vqa, etc.) are inherited from VADAR.

Key Contributions

  • Closed-loop program-tool-program cycle: each phase feeds the other, enabling continuous self-improvement without human intervention.
  • Dual-library architecture separating experiential memory (\(\mathcal{E}\)) from callable abstractions (\(\mathcal{T}\)), with both growing across iterations.
  • Two-stage tool validation (execution + VLM correctness judge) that enforces reliability before any tool enters \(\mathcal{T}\).
  • Tool Library maintenance via periodic merging, keeping \(\mathcal{T}\) concise (61 total tools created → 11 active after merging).
  • Demonstrated zero-shot generalization: libraries built on Omni3D-Bench transfer to 3DSR-Bench, SpatialSense, and VG-Bench without modification.

Results

  • Omni3D-Bench overall accuracy: TVP 33.3% vs. VADAR 29.9% (+11%), GPT-4o 27.2% (+22%).
  • Float calculation accuracy: TVP 36.5% MRA vs. GPT-4o 8.2%, VADAR 35.5% MRA — largest gain on geometrically demanding question type.
  • Tool reuse: 36.3% of TVP programs use only abstracted tool calls vs. 5.8% for VADAR; TVP tools used 5× more frequently as core dependency.
  • Program quality gains from abstraction: +3.4% accuracy when switching programs to newly created tools; median cyclomatic complexity drops from 3.0 to 1.0 across iterations.
  • Iteration trajectory: TVP (Full) improves 31.3% → 31.9% → 33.3%; TVP (Example-Lib-Only) plateaus at 31.7% → 31.5% → 31.5%.
  • Hard-question scaling: On hardest questions (complexity 7–10), TVP (Full) starts −4.5% below Example-Lib-Only in iteration 1 but reaches +6.7% above it by iteration 3.
  • Backbone robustness: Performance scales with Qwen2.5-Coder-Instruct model size; clear scaling trend observed.
  • Zero-shot transfer: TVP outperforms all baselines on 3DSR-Bench, SpatialSense, VG-Bench without testset-specific modification.

Limitations

  • Depends on GPT-4o for program generation, quality judgment, and tool abstraction — not lightweight at inference time.
  • The initial 5 basic tools (GroundingDINO, UniDepth, etc.) form a fixed foundation; tool quality is bounded by these atomic operations.
  • Evaluated only on 3D spatial reasoning; generality to other visual reasoning domains (e.g., relational reasoning, temporal reasoning) is not demonstrated.
  • Example Library admission requires executing programs on the dataset before test time, making the method transductive rather than purely inductive/zero-shot.
  • Merging heuristics and similarity thresholds (\(\tau_\text{sim}\), \(\tau_\text{cluster}\), \(\tau_\text{potential}\), \(\tau_\text{correct}\)) are hyperparameters with ablation not fully explored in the visible text.

Relevance to Vision-Language Models

TVP directly addresses a known failure mode of VLMs: precise geometric reasoning requiring multi-step arithmetic, where even GPT-4o achieves only 8.2% on floating-point spatial queries. The framework treats VLMs as program generators and judges within a structured loop, rather than as monolithic reasoners, and demonstrates that compositional tool-use dramatically outperforms spatial fine-tuning (SpaceMantis, SpatialBot-3B show no improvement over generic VLMs). For VLM researchers, TVP establishes a principled alternative to prompt engineering or fine-tuning for precision-demanding tasks: evolving a domain-specific tool API from experience. The zero-shot transfer result also suggests that experience-grounded tool libraries encode generalizable spatial reasoning abstractions beyond any single benchmark's distribution.