Skip to content

Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; agentic jigsaw interaction improves fundamental VLM perceptual reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AGILE trains vision-language models to solve jigsaw puzzles through iterative code-generating agent loops, using the structured puzzle task as a scalable proxy for learning visual perception and reasoning. Starting from a 9.5% baseline, a 7B VLM reaches 82.8% accuracy on 2×2 jigsaw tasks and transfers +3.1% average improvement across 9 general vision benchmarks. The key insight is that programmatically generated jigsaw data sidesteps the chronic scarcity of high-quality multimodal RL data.

Problem

Current VLMs perform near-randomly on simple jigsaw tasks (e.g., 9.5% on 2×2), exposing fundamental weaknesses in spatial perception and structured reasoning. Improving these abilities via RL requires large-scale, high-quality multimodal training data, which is expensive to annotate by humans and quality-limited when synthesized from closed-source APIs.

Method

AGILE casts jigsaw solving as a multi-turn agentic loop: at each step the policy model (Qwen2.5-VL-7B) generates Python code calling one of four environment APIs—Swap (exchange two tiles), Observe (view current state), Crop (extract a region), Zoom (magnify a region)—receives a rendered image as feedback, and iterates until the puzzle is solved or the step budget \(T\) is exhausted.

Training has two stages:

  1. Cold-start SFT: 1.6K expert trajectories collected from Gemini 2.5 Pro (filtered to ground-truth-correct, manually verified, balanced over step counts 4–8 and action types) are used for supervised fine-tuning to bootstrap instruction-following and code generation.

  2. GRPO RL: 15.6K images partitioned into 2×2 grids are used for Group Relative Policy Optimization. The reward combines:

  3. Accuracy \(R_{acc} \in \{0,1\}\): all tiles correctly placed
  4. Format \(R_{format} \in \{0,1\}\): correct <think>, <code>, <answer> structure
  5. Step penalty \(R_{step} = \lambda \cdot \bigl(\mathbf{1}_{R_{acc}=1} \cdot \text{stepnum} + \mathbf{1}_{R_{acc}=0} \cdot \text{stepmax}\bigr)\) with \(\lambda=-0.05\)
  6. Total: \(R = 0.8 R_{acc} + 0.2 R_{format} + 1.0 R_{step}\)

Jigsaw data are fully synthetic (programmatic shuffling with known ground truth), making the dataset arbitrarily scalable without human annotation or API cost.

Key Contributions

  • AGILE framework: formulates jigsaw solving as a stepwise agent–environment loop with code-generated actions and visual feedback, driving iterative refinement of perception and reasoning.
  • Scalable RL data generation: purely programmatic jigsaw synthesis with controllable difficulty (grid size \(m\), number of pre-placed tiles) eliminates dependence on human annotation or closed-source models.
  • Two-stage training pipeline: cold-start SFT on 1.6K Gemini-curated trajectories followed by GRPO RL on 15.6K images.
  • Empirical evidence that proxy-task RL transfers: jigsaw training improves performance on 9 general vision benchmarks with no task-specific data for those benchmarks.

Results

  • Jigsaw 2×2 (Acc): Qwen2.5-VL-7B baseline 9.5% → AGILE 82.8%; Gemini 2.5 Pro 46.4%, Qwen2.5-VL-72B 27.4%
  • Jigsaw 2×2 (Score): 29.4% → 89.0%
  • Jigsaw 3×3 (Acc): 0.4% → 20.8%, surpassing Gemini 2.5 Pro (14.6%) and Qwen2.5-VL-72B (3.9%)
  • Jigsaw 3×3 (Score): 31.1% → 62.1%
  • General vision (9 benchmarks average): +3.1% over Qwen2.5-VL-7B baseline (62.1% → 65.2%)
  • HRBench4K: +4.2%, HRBench8K: +5.2%, VStarBench: +4.2%, MME-RealWorld: +3.8%
  • Data scaling: jigsaw accuracy rises monotonically from 22.0% to 82.8% as training set grows; HRBench4K +2.0%, RealWorldQA +1.8% from scaling alone
  • Jigsaw vs. QA data: jigsaw-trained models outperform QA-trained models on general vision benchmarks; combining 10K jigsaw + 10K QA exceeds either alone

Limitations

  • RL training conducted only on 2×2 grids; 3×3 generalization is partial (20.8% Acc vs. 82.8% on 2×2), suggesting the skill transfer becomes harder at higher grid complexity.
  • Cold-start trajectories still depend on Gemini 2.5 Pro for bootstrapping, introducing a closed-source dependency for the initial SFT phase.
  • Base model constrained to Qwen2.5-VL-7B; it is unclear how the framework scales to architecturally different or larger VLMs.
  • General vision improvement (+3.1%) is modest relative to the large gains on the proxy task, leaving open how much of the gain is perception vs. format/reasoning artifacts.
  • Evaluation of general benchmarks is single-turn only (VLMEvalKit), preventing the model from using its learned multi-turn interaction capability on downstream tasks.

Relevance to Agentic AI / LLM Agents

AGILE is a direct instantiation of the tool-using, environment-interacting agent paradigm applied to vision: the model generates executable code, receives grounded feedback, and iterates—a pattern central to agentic LLM research. It demonstrates that a structured, verifiable proxy environment (jigsaw puzzles) can serve as a scalable RL training ground, sidestepping the data-bottleneck that plagues real-world agentic training. The GRPO-based interactive rollout shares architecture with recent code-agent and tool-calling RL work (e.g., DeepSeek-R1 extended to VLMs), and the reward decomposition (accuracy + format + efficiency) mirrors reward shaping strategies debated across the agentic RL literature. For researchers tracking LLM agents, the paper offers a concrete recipe for generating unlimited verifiable agent-interaction data from any structured puzzle domain.