Sherlock: Self-Correcting Reasoning in Vision-Language Models¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Sherlock is a three-stage training framework (SFT cold-start → offline preference learning → online self-improvement) that teaches reasoning VLMs to perform response-wise self-correction at the trajectory level. Built on Llama3.2-Vision-11B, it achieves 64.1 average accuracy (direct) and 65.4 (after self-correction) across eight multimodal benchmarks using only 20k annotated examples — outperforming baselines trained on 5–13× more data.

Problem¶

Current reasoning VLMs (SFT- and RL-trained alike) cannot self-correct: step-wise "aha moments" appear in fewer than 10% of trajectories, and response-wise correction via prompts or external critics either fails to improve or actively degrades accuracy. Existing approaches are also heavily data-dependent (100k–260k annotated samples) and require accurate verifiers or large-scale critique data, limiting generalization beyond math/VQA domains.

Method¶

Sherlock teaches response-wise self-correction through three stages:

SFT cold-start: A pairwise loss jointly trains direct generation and self-correction on 20k samples (10k for base R0 VLM; 10k for preference pairs where the preferred response Y^w comes from ground-truth annotations and the rejected Y^l from the weaker R0 VLM).
Offline trajectory-level preference training: Rather than rewriting entire responses, the model learns to correct only the erroneous suffix Y^l_≥i, given a shared prefix Y_<i. Negative responses are constructed by truncating at a random step i and regenerating from a visually-perturbed image (Gaussian noise injection). A sample-specific dynamic β — inversely scaling with truncation position i and perturbation strength ε — stabilizes training by assigning larger β (more conservative KL constraint) to pairs with wider quality gaps. The full loss combines a novel trajectory-level self-correction loss (Eq. 6) with standard DPO.
Online iterative self-improvement: Using 5k unlabeled inputs per iteration, the model generates four rounds of self-correction (Y^1 → Y^4). If a^2 = a^3 = a^4 (self-consistency), Y^4 becomes the preferred response; the rejected response is formed by perturbing Y^1's suffix with visual noise. Two iterations (Iter1, Iter2) refine both reasoning and self-correction without any ground-truth labels.

Key Contributions¶

Empirical audit showing that current SFT- and RL-trained reasoning VLMs cannot self-correct in either step-wise or response-wise modes.
Trajectory-level self-correction objective: corrects only the erroneous suffix Y_≥i rather than the full response, yielding cleaner learning signals.
Visual perturbation-based preference data construction that creates controllable quality gaps without requiring a verifier.
Dynamic β that adapts per sample to the magnitude of the quality gap, stabilizing DPO-style preference optimization.
Online self-improvement loop requiring zero ground-truth labels after the initial 20k examples.
Integration with MM-Verify: sequential self-correction + verifier reduces GPU usage by 40% (13.2 → 8.7 A100 hours on MathVista) while improving accuracy (54.0 → 55.9).

Results¶

Sherlock Iter2 achieves 64.1 (direct) / 65.4 (after 3-round self-correction) average across 8 benchmarks (MMBench, MMVet, HallusionBench, MMMU, MMStar, AI2D, MathVista, MME).
Outperforms LLaVA-CoT (63.2, 100k data), Mulberry (63.9, 260k), LlamaV-o1 (63.4, 175k) — all same backbone — while using <20% of annotated data.
Baselines degrade after self-correction: LLaVA-CoT −0.2, Mulberry −0.1, LlamaV-o1 −15.2 (format breakage).
Sherlock's self-correction gains grow across training stages: SFT +0.8↑, Offline +1.2↑, Iter1 +1.2↑, Iter2 +1.3↑.
With strong critic Qwen2.5-VL-7B: average 64.9 (+0.8↑); with self-correction alone: 65.4 (+1.3↑).
Sequential self-correction + MM-Verify: 55.9 accuracy at 8.7 GPU-hours vs. parallel vote + verify at 55.1 accuracy at 40.2 GPU-hours on MathVista.
Ablations confirm: trajectory-level (random i) > full-response correction (i=1); dynamic β > static β; joint DPO + SC loss > either alone.

Limitations¶

Self-correction is response-wise only; step-wise in-trajectory correction (which would be more compute-efficient) is left to future work.
Online self-improvement relies on self-consistency filtering (a^2 = a^3 = a^4) as a proxy for correctness — unreliable for open-ended or ambiguous questions without verifiable answers.
Visual perturbation as the sole negative-sample construction strategy may not cover all failure modes (e.g., reasoning errors unrelated to visual grounding).
Evaluation is limited to eight benchmarks; generalization to non-mathematical, non-VQA domains (e.g., document understanding, embodied tasks) is asserted but not demonstrated.
Still depends on 20k CoT-annotated seed examples from LLaVA-CoT; fully unsupervised bootstrapping is not achieved.

Relevance to Vision-Language Models¶

Sherlock directly addresses a recognized brittleness in reasoning VLMs: error propagation through multi-step chains of thought. By framing self-correction as a learnable, trajectory-level preference optimization problem — rather than a prompting trick — it provides a principled path toward VLMs that can iteratively refine their own multimodal reasoning. The data efficiency finding (20k random samples matching models trained on 260k curated examples) is significant for the broader community, suggesting that self-correction as an auxiliary objective is a strong regularizer for general reasoning. The online self-improvement stage, which requires no ground-truth labels, connects directly to scalable post-training paradigms being explored for LLMs and positions self-correction as a mechanism for autonomous capability growth in VLMs.