Waking Up Blind: Cold-Start Optimization of Supervision-Free Agentic Trajectories for Grounded Visual Perception

🕒 Published (v1): 2026-04-19 15:06 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Cold-start agentic VLM grounding without expensive supervised trajectory tuning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SPECTRA is a supervision-free framework that trains small vision-language models (SVLMs, ≤7B) as agentic tool-using controllers via cold-start reinforcement learning, without any human-annotated trajectories. It enforces a topological rollout structure (reason→tool→observe→perceive→reason→answer) and optimizes a multi-objective reward via GRPO. On four multimodal QA benchmarks plus out-of-distribution MMMU-Pro, SPECTRA improves task accuracy by up to 5% and tool efficiency by 9% over the strongest VERL-based agentic baseline.

Problem

SVLMs used as agentic controllers suffer from visual brittleness and poor tool orchestration; closing this gap normally requires expensive supervised trajectory data distilled from large models. Existing RL approaches also lack a principled way to evaluate tool efficacy without ground-truth tool annotations.

Method

SPECTRA fine-tunes Qwen2.5-VL (3B and 7B) using LoRA adapters (vision encoder frozen) via Group Relative Policy Optimization (GRPO) with no supervised demonstrations.

Soft Structured Multi-turn Rollouts: Every trajectory is constrained to the topology \(\tau = \langle \text{reason} \to \text{tool} \to \text{obs} \to \text{percep} \to \text{reason} \to \text{ans} \rangle\). The structural reward \(R_{\text{struct}}(\tau) = \alpha \cdot \gamma^{\varphi(\tau)}\) (with \(\alpha=2.0\), \(\gamma=0.75\)) decays as deviations from the optimal topology increase.

Multi-objective reward (Eq. 5): \(R_{\text{total}} = \lambda_1 R_{\text{corr}} + \lambda_2 R_{\text{struct}} + \lambda_3 R_{\text{tool}} + \lambda_4 R_{\text{term}}\), covering task correctness, structural integrity, tool diversity/validity, and answer termination. A per-tool saturation cap \(\kappa\) and global diversity cap \(\eta\) prevent reward hacking via excessive or repetitive tool calls.

Cold-start GRPO: No reference policy from human preferences; group-relative advantage normalization \(\hat{A}_i = \frac{R_{\text{total}}(\tau_i) - \mu}{\sigma + \delta}\) over \(G\) sampled trajectories per input stabilizes learning when the policy must deviate substantially from its initialization.

Tool Instrumental Utility (TIU): A new evaluation metric combining Tool Execution Reliability (TER), Task-Tool Alignment Coefficient (TTAC, point-biserial correlation between tool use and success), and Tool Selectivity Score (TSS, KL-divergence from uniform): $\(\text{TIU} = \text{TER} \times \frac{1+\text{TTAC}}{2} \times \tanh(\text{TSS})\)$

The toolbox contains four primitives: Image Captioning, Object Detection, OCR, Visual Perception.

Key Contributions

• SPECTRA framework: supervision-free cold-start RL for agentic SVLMs using structured multi-turn rollouts enforced as topological constraints.
• Soft Structured Multi-turn Rollouts: a topological sequence constraint (Eq. 2) that forces tool evidence to precede synthesis, grounding visual reasoning without labeled data.
• TIU metric: composite, ground-truth-free metric for tool efficacy, integrating reliability, alignment, and selectivity.
• Empirical demonstration on four in-distribution benchmarks (AI2D, TQA, ScienceQA, OK-VQA) plus OOD (MMMU-Pro) with statistical validation (\(p = 0.0019\)).

Results

• Task accuracy (7B, avg. in-distribution): SPECTRA 77.8% vs. closest VERL baseline 73.4% (+4.4 pp); gains of 4.8 pp on ScienceQA and 5.0 pp on OK-VQA.
• OOD (MMMU-Pro): SPECTRA 7B 46.7% vs. VERL baseline 44.3% (+2.4 pp); vs. GPT-4o 61.8%.
• Task accuracy (3B): SPECTRA 63.9% avg. in-distribution vs. VERL 3B baseline 60.3%.
• TIU (7B, mean): 44.66% vs. baseline 35.63% (+25.3% relative).
• TER (7B, mean): 88.69% vs. baseline 77.30% (+11.4 pp).
• TSS (7B, mean): 2.98 vs. baseline 2.05 (better selectivity, avoids mode collapse).
• TTAC shifts from negative mean (−0.003) to near-zero positive (0.009), indicating self-discovered policies correlate tool use with success.
• Ablation (Table 3, 7B): removing any single reward component (\(R_{\text{corr}}, R_{\text{struct}}, R_{\text{tool}}, R_{\text{term}}\)) consistently degrades average accuracy (drops of 1.3–2.1 pp), confirming each component is load-bearing.

Limitations

• TTAC remains near zero even after training, indicating that cold-start conditions make aligning tool use with task success genuinely difficult without supervision.
• Despite recovery mechanisms, SPECTRA still occasionally hallucinates undefined tool tags and enters generation loops before recovering (Figure 6).
• Evaluated only on MCQ-style benchmarks; generalization to open-ended generation or longer-horizon tasks is untested.
• Only four tools in the action space; the approach's scalability to richer, more diverse toolsets is unvalidated.
• Results are limited to Qwen2.5-VL backbone; cross-architecture generalization is not demonstrated.

Relevance to Vision-Language Models

SPECTRA directly addresses a core bottleneck for deploying small VLMs as practical agents: the need for expensive supervised trajectory data to teach tool-grounded visual reasoning. By combining structured rollout constraints with cold-start GRPO, it offers a pathway to efficient, self-improving multimodal agents that can be trained from environment feedback alone—a relevant design principle as the field moves toward autonomous VLM-based systems. The TIU metric fills a genuine evaluation gap, since standard accuracy metrics cannot capture whether tool use is contributing purposefully to correct answers. This work complements trajectory-distillation approaches (T3-Agent, MLLM-Tool) by showing competitive gains without the teacher model overhead.