Skip to content

GAVEL: Grounded Caption Error Verification and Localization

🕒 Published (v1): 2026-06-25 12:00 UTC · Source: Arxiv · link

Why this paper was selected

VLM hallucination detection+localization; directly useful for grounded VLM eval pipelines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GAVEL is a benchmark and dataset for jointly evaluating hallucination detection, natural-language explanation, and visual grounding (bounding-box localization) in VLMs. It introduces 35k training and 5k test image–caption pairs with human-annotated explanations and bounding boxes. Even GPT-5 achieves only 34.1% coarse grounding accuracy (A@0.3), and a supervised fine-tuned baseline outperforms all zero-shot models on localization.

Problem

Prior hallucination benchmarks provide only coarse binary correctness judgments or text-only corrections, offering no insight into why a mismatch occurs or where in the image it is grounded. This limits interpretability and diagnostic utility for building reliable VLMs.

Method

GAVEL frames the problem as a three-output task: given an image–text pair, a model must (1) detect hallucination, (2) generate a natural-language explanation of the discrepancy, and (3) predict a bounding box over the corresponding image region. The dataset is built via a semi-automatic pipeline—VLMs or text-to-image models generate candidate pairs, GPT-5 mini filters for low-alignment pairs, and human annotators provide final explanations and bounding boxes through three-stage quality verification. The supervised baseline extends VisionLLM with 1025 discrete coordinate tokens \(\tau_i\) for \(i \in [-512, 512]\), serializing bounding boxes as autoregressive token sequences via coordinate quantization: $\(q(v, S) = \text{round}\!\left(512 \cdot \text{clip}\!\left(\frac{v - S/2}{S/2}, -1, 1\right)\right).\)$ Explanation quality is scored by GPT-5-as-judge on a 1–5 scale (reported 0–100).

Key Contributions

  • GAVEL task: unified benchmark requiring hallucination detection, explanation, and visual grounding simultaneously.
  • Dataset: 35,249 training pairs and 5,606 test pairs with human-annotated bounding boxes and explanations, sourced from both image-to-text and text-to-image pipelines; covers 10+ fine-grained hallucination categories (object, spatial, color, attribute, action, state, quantity, etc.).
  • Evaluation protocol: IoU-based grounding accuracy at A@0.3/0.5/0.7 plus LLM-scored explanation quality, with category-level breakdowns.
  • Supervised baseline: VisionLLM fine-tuned with coordinate-token autoregression, demonstrating that the dataset contains learnable grounding signals.

Results

  • GPT-5 (best explanation): A@0.3=34.1, A@0.5=17.4, A@0.7=6.5; LLM Avg=70.
  • Qwen3-VL-30B (best strict grounding): A@0.3=30.8, A@0.5=22.8, A@0.7=16.4; LLM Avg=62.
  • 8B open-source models are substantially weaker: InternVL3.5-8B A@0.3=17.3, LLM Avg=44; Qwen3-VL-8B A@0.3=15.1, LLM Avg=54.
  • Supervised baseline vs. base Qwen3-VL-8B (zero-shot): A@0.3 improves 15.1→42.6, A@0.5 12.1→30.2, A@0.7 9.1→18.4, LLM Avg 54→58.
  • GPT-5-mini exhibits bounding-box over-sizing bias; no systematic size bias in other models, but localization errors are not reducible to box-size miscalibration alone.

Limitations

  • Training exploration is preliminary; no systematic study of training strategies, data scaling, data mixture, or architecture choices.
  • LLM-as-judge evaluation (GPT-5) introduces its own reliability and cost dependencies.
  • Dataset covers a specific distribution of hallucination types; generalization to other VLM failure modes (e.g., reasoning errors, OCR errors) is not assessed.
  • Bounding-box localization assumes hallucinations map to single contiguous regions, which may not hold for relational or compositional errors.

Relevance to Vision-Language Models

GAVEL directly addresses a core open problem in VLM evaluation: moving from coarse binary hallucination detection to interpretable, localized diagnoses. For researchers tracking VLMs, it provides both a diagnostic benchmark that exposes gaps in state-of-the-art closed-source models (GPT-5 peaks at A@0.3=34.1%) and a training dataset that demonstrably improves grounding without task-specific architectures. The coordinate-token autoregression approach shows that spatial grounding can be unified with language generation in a single autoregressive framework, a direction relevant to grounded VLM training more broadly. GAVEL also bridges hallucination detection with referring expression grounding, connecting to lines of work on RLHF-based faithfulness and interpretable multimodal alignment.