DiVE-k: DIFFERENTIAL VISUAL REASONING FOR FINE-GRAINED IMAGE RECOGNITION

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Ram Nevatia (USC); ICLR 2026; differential visual reasoning for fine-grained recognition

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DiVE-k addresses the failure of Large Vision-Language Models (LVLMs) to perform fine-grained visual discrimination by reframing recognition as a multiple-choice question (MCQ) using the model's own top-\(k\) predictions as confusing alternatives. RL training via GRPO on this self-constructed MCQ dataset compels the model to reason differentially about visually similar categories. On five fine-grained benchmarks, DiVE-k surpasses the strongest RL baseline (ViRFT) by +6.2% Harmonic Mean in the base-to-novel zero-shot setting.

Problem

LVLMs rely on coarse, salient visual attributes shared across related categories, leading to a large gap between Pass@1 and Pass@K accuracy in fine-grained recognition. Existing RL-based fine-tuning (e.g., ViRFT) uses exact string-match rewards that are brittle to name variants, encourage memorization of training class names, and fail to elicit the attribute-level discriminative reasoning needed to generalize to unseen ("novel") categories.

Method

DiVE-k is a two-step framework built on GRPO:

Step 1 — Offline option mining. For each training image \((I, q)\), \(K=20\) responses are sampled from the base policy \(\pi_\theta\). The \(k=\min(5, |C|)\) most frequent predicted categories form an option set \(O_{\text{top-}k}\); if the ground-truth \(\hat{c} \notin O_{\text{top-}k}\), the least-frequent candidate is replaced by \(\hat{c}\). Options are randomly shuffled to avoid position bias, yielding a dataset \(\mathcal{D} = \{(I, q, O_{\text{enum}}, \hat{a})\}\). Trivial examples where only one (correct) category is ever predicted are excluded as hard-negative mining.

Step 2 — RL training with GRPO. The policy is trained on MCQ prompts using GRPO with group advantage: $\(A_i = \frac{r_i - \text{mean}(\{r_j\})}{\text{std}(\{r_j\}) + \delta}\)$ The reward \(r = \lambda_f r_{\text{format}} + \lambda_m r_{\text{mcq}}\) combines a binary MCQ correctness signal (\(r_{\text{mcq}} \in \{0, 1\}\), checking predicted option index against \(\hat{a}\)) with a format compliance reward. The GRPO objective also includes a KL penalty \(\beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})\) for stability.

Inference mirrors training: the trained policy first generates \(K\) rollouts to construct options, then is re-prompted with the MCQ to select the answer. No ground-truth is injected at inference.

Key Contributions

• Self-referential MCQ construction: using the model's own top-\(k\) predictions as distractors forces differential reasoning without external ontologies.
• Verifiable, non-brittle reward: MCQ index-matching replaces fragile string-match, accommodating scientific vs. common name variants.
• Demonstrated generalization: base-to-novel HM improvements over ViRFT across five fine-grained benchmarks and three settings (standard, mixed-domain, few-shot).
• Hard-negative filtering: trivially-correct samples are excluded, focusing training on genuinely ambiguous cases.

Results

Zero-shot base-to-novel generalization (5 datasets, Harmonic Mean averages): - DiVE-k avg HM: 79.8 (base 80.8 / novel 78.8) - vs. QWEN2.5-VL-7B: +10.1% HM (+11.9 base, +8.3 novel) - vs. ViRFT: +6.2% HM (+7.8 base, +4.6 novel) - vs. ViRFT using DiVE-k inference pipeline (ViRFT*): +2.9% HM - Largest per-dataset gains: CUB +14.9 HM, Oxford Flowers +8.5 HM - DiVE-k matches Gemini2.5-flash-lite (80.0 HM) and surpasses Grok-4-fast (71.8 HM); GPT-5-mini (83.4 HM) leads

Mixed-domain zero-shot base-to-novel (single model trained on all base sets): - DiVE-k avg HM: 78.7 vs. ViRFT 75.0 (+3.7 HM); vs. QWEN2.5-VL-7B 69.3 (+9.4 HM)

4-shot classification: - DiVE-k improves average accuracy by +7.73% over ViRFT

SFT baseline pathology: - SFT (Full) collapses on novel categories: avg novel accuracy drops 33.5% relative to the base model, avg HM falls to 49.8

Limitations

• Two-step inference increases compute: \(K\) rollouts are needed at test time before the MCQ re-prompt, roughly doubling inference cost.
• Mild regression on OxfordPets vs. ViRFT, attributed to the larger option set causing errors in the second step; smaller \(k\) mitigates this.
• Options are sourced from a fixed base model's distribution; if the base model rarely generates the correct category in \(K\) rollouts, it is excluded from the option set at inference (ground-truth is not injected during test).
• All experiments use a single 7B backbone (QWEN2.5-VL-7B); scaling behavior to larger or different architectures is not evaluated in the main paper.
• Performance on Aircraft and Cars remains below proprietary models, suggesting limits at very high inter-class similarity.

Relevance to Vision-Language Models

DiVE-k directly tackles a known weakness of LVLMs — that their language knowledge is not effectively mobilized for fine-grained visual disambiguation — by coupling RL with a self-supervised MCQ signal derived from the model's own uncertainty. This is significant for the LVLM fine-tuning literature because it replaces brittle string-match rewards (a widespread design choice since ViRFT/DeepSeek-R1) with a format-agnostic, index-based reward that naturally elicits Chain-of-Thought differential reasoning. The result connects to broader work on how Pass@K diversity can be converted into a training signal rather than just an evaluation metric, and offers a template for improving LVLM specialization in any closed-world recognition domain without requiring human-annotated reasoning traces.