Self-Aug: Query and Entropy Adaptive Decoding for Large Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; query-adaptive entropy-based contrastive decoding mitigates LVLM hallucination

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Self-Aug is a training-free decoding strategy for Large Vision-Language Models (LVLMs) that reduces hallucinations by (1) using the model's own parametric knowledge to select a query-specific visual augmentation for contrastive decoding, and (2) replacing the static Adaptive Plausibility Constraint with an entropy-driven dynamic threshold. Evaluated across five LVLMs and seven benchmarks, Self-Aug achieves up to 18.78% average gain over multinomial sampling.

Problem

Existing Visual Contrastive Decoding (VCD) methods apply query-agnostic visual augmentations, ignoring the semantic intent of the text query—asking "what color is the coat?" and "solve this math problem" require entirely different visual disruptions. Additionally, the Adaptive Plausibility Constraint (APC) in CD-based methods is confidence-agnostic: it truncates the candidate token set using only the maximum logit value, discarding useful signal from the full logit distribution and performing unreliably under high model uncertainty. VACoDe's query-aware approach relies on first-token L2 divergence, which fails for open-ended generation and multi-step reasoning.

Method

Self-Augmentation Selection (SAS): A structured meta-prompt \(P\) is fed to the LVLM itself, asking it to reason over the query and then select the most semantically disruptive augmentation from a predefined set {random crop, random mask, noise addition, color inversion, horizontal flip, vertical flip}. The prompt includes augmentation definitions, chain-of-thought scaffolding, and in-context learning examples. The model's output is parsed to extract a reasoning trace \(r\) and a choice \(c\): $\((r, c) = g(f_\theta(P, x)), \quad v' = A(c, v).\)$ SAS uses greedy decoding for efficiency and determinism.

Sparsity Adaptive Truncation (SAT): Replaces APC's fixed \(\beta\) with a per-timestep threshold derived from the Shannon entropy of the expert logit distribution: $\(\beta_t^{SAT} = \sigma\!\left(-\gamma \sum_{i} p_i \log_2 p_i\right), \quad \gamma < 0,\)$ where \(\sigma\) is the sigmoid. High entropy (low confidence) yields a lower \(\beta\), expanding the candidate set; low entropy (high confidence) yields a higher \(\beta\), restricting it. The sigmoid bound keeps \(\beta_t^{SAT} \in (0, 0.5]\), preventing collapse to greedy decoding. The final contrastive logit is: $\(l_{CD}(y_t) = \begin{cases}(1+\alpha)\,l - \alpha\,l' & \text{if } y_t \in V_{SAT} \\ -\infty & \text{otherwise}\end{cases}\)$

Key Contributions

• SAS prompting: leverages LVLM's intrinsic world knowledge for query-aligned visual augmentation selection, replacing heuristic first-token divergence.
• SAT thresholding: entropy-aware dynamic plausibility constraint that utilizes the full logit distribution instead of only the maximum value.
• Training-free, architecture-agnostic: integrates with any LVLM without modification or additional training.
• Comprehensive evaluation: five LVLMs (LLaVA-1.5-7B/13B, Qwen-VL, InstructBLIP, Qwen3-VL-8B) across seven benchmarks covering both discriminative and open-ended generative settings.

Results

• Self-Aug achieves up to +18.78% Avg. ∆ over multinomial sampling baseline (InstructBLIP), outperforming VCD (+12.33%) and VACoDe (+10.98%).
• LLaVA-1.5-7B: +14.32% vs. VCD +10.86%, VACoDe +9.52%; MME-P: 1431 vs. 1278 (baseline).
• LLaVA-1.5-13B: +11.59% vs. VCD +6.33%, VACoDe +8.03%; MME-P: 1462 vs. 1352 (baseline).
• InstructBLIP: +18.78% vs. VCD +12.33%, VACoDe +10.98%; MME-P: 1199 vs. 974 (baseline).
• Qwen-VL: +17.09% on generative benchmarks vs. VCD +9.21%, VACoDe +10.47%.
• Qwen3-VL-8B (stronger model): gains are smaller (+2.25% vs. +1.00% VCD, +2.07% VACoDe), with MMHal-Bench score 4.67 and LLaVA-Bench 121.88.
• MMHal hallucination rate reduced on LLaVA-1.5-7B from 0.65 (baseline) to 0.59 with Self-Aug.

Limitations

• SAS requires one additional greedy forward pass per query, adding latency proportional to augmentation-selection response length.
• Augmentation set is fixed to six types; combinations of augmentations are not explored.
• Gains diminish substantially on stronger models (Qwen3-VL-8B: ~2% vs. ~18% on InstructBLIP), suggesting limited headroom as model capability scales.
• SAS relies on the model correctly interpreting the meta-prompt; failure cases from incorrect augmentation selection are not systematically analyzed.
• Ablation studies are limited to LLaVA-1.5-7B on MME-P; cross-model generalization of each component is not fully characterized.

Relevance to Vision-Language Models

Self-Aug directly targets LVLM hallucination—one of the field's most persistent failure modes—without any additional training, making it immediately applicable to any deployed LVLM. The SAS component advances the VCD line of work by replacing arbitrary visual corruption with model-reasoned, query-conditioned augmentation, which is a principled step toward semantics-aware decoding. SAT addresses a known weakness of APC and is independently applicable to other CD-based decoding methods. The finding that gains shrink on stronger models (Qwen3-VL-8B) provides evidence that decoding-level interventions may become less critical as underlying model alignment improves, which is useful context for the broader VLM research trajectory.