Skip to content

Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Adversarial referring expression tasks expose gaps in VLM visual reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Ref-Adv is a new Referring Expression Comprehension (REC) benchmark designed to close the shortcut-exploitation loophole in RefCOCO/+/g, where models achieve >90% accuracy without genuine visual or textual reasoning. It pairs linguistically complex, minimally sufficient expressions with hard distractors (partially matching but incorrect instances), exposing severe performance gaps in state-of-the-art MLLMs.

Problem

Classic REC benchmarks (RefCOCO, RefCOCO+, RefCOCOg) suffer from three compounding weaknesses: (1) expressions average only 3.6 words, limiting language complexity; (2) most images contain ≤1 same-category distractor, reducing visual discrimination demand; and (3) "grounding shortcuts" arise when overspecified expressions are paired with too few distractors, allowing models to localize targets by matching any single descriptor rather than reasoning over the full expression. These shortcuts enable near-saturated performance (~90%+ Acc@0.5) without testing genuine reasoning.

Method

Ref-Adv is constructed via a four-stage LLM-assisted pipeline on COCO and OpenImages v7: 1. Input Preparation: Filter images to retain only those with ≥3 same-category instances (ensuring hard distractor presence); apply Set-of-Marks number tagging to candidate instances. 2. Similarity Judgement: Prompt GPT-4o to identify the most similar pair (target + hard distractor as group A; remaining distractors as group B) and elicit both group-level and within-group discriminating attributes. 3. Expression Generation: Compose minimally sufficient referring expressions from a curated subset of discriminators, using either affirmative descriptors of the target or negation of the hard distractor's attributes—deliberately avoiding redundant descriptors. 4. Human Verification: Three annotators verify expression correctness, unambiguity, and the existence of hard distractors; pairs require unanimous agreement (18.7% keep rate for LLM-authored items).

A smaller human-authored subset is also collected. Quality is validated via: (a) model bias tests (replacing expressions with "the one"), (b) bag-of-words word-order ablations, and (c) single-descriptor-deletion sufficiency tests.

Key Contributions

  • Ref-Adv benchmark: 5,000 instances across 2,833 images with avg. expression length 11.5 words, avg. 4.01 distractors/image, 21.25% negation ratio—substantially harder than RefCOCO/+/g on all axes.
  • A two-stage LLM-authored pipeline (similarity judgement → minimally sufficient expression generation) that structurally prevents overspecification and shortcut grounding.
  • Ablation protocols (bag-of-words word-order randomization, single-descriptor deletion, fixed-prompt bias test) that empirically verify reasoning necessity.
  • Comprehensive evaluation of 13 contemporary MLLMs (Qwen2.5-VL, InternVL-3, Gemini 2.5-Flash/Pro, GPT-4o, Claude-3.5 Sonnet, GLM-4.5V, CogVLM-Grounding) with and without Chain-of-Thought (CoT).
  • In-depth failure analysis showing models predominantly select the hard distractor, pinpointing failures in visual perception and full-expression understanding.

Results

  • All models drop markedly on Ref-Adv vs. RefCOCO/+/g; best Acc@0.5 is GPT-4o+CoT+SoM at 63.7%, compared to 90%+ on RefCOCO.
  • Qwen2.5-VL-72B: 54.1% (direct) / 58.3% (CoT) on Ref-Adv vs. 92.7% on RefCOCO.
  • InternVL-3-78B: 54.6% (direct) / 58.4% (CoT) on Ref-Adv.
  • Gemini 2.5-Flash: 50.6% (direct) / 59.4% (CoT); Gemini 2.5-Pro: 51.9% / 59.1%.
  • Claude-3.5 Sonnet (SoM): 40.8% / 45.2% (CoT).
  • CoT helps on Ref-Adv (+4.2% for Qwen2.5-VL-72B) but hurts slightly on RefCOCO/+/g (−1.4% to −3.4%), consistent with the heavier reasoning demand of Ref-Adv.
  • Bag-of-words ablation: Ref-Adv drops −16.8% (Qwen2.5-VL-72B) vs. −9.9% on RefCOCO, confirming stronger textual reasoning dependence.
  • Single-descriptor deletion drops Ref-Adv by −6.4% (Qwen) vs. −4.7% on RefCOCO, confirming fewer grounding shortcuts.
  • Fixed-prompt ("the one") bias gap: RefCOCO ~35% vs. Ref-Adv ~22%, showing less benchmark contamination.
  • Models in the ≥7 distractor group show larger drops (e.g., Qwen2.5-VL-72B+CoT: −2.7 pp) than 4–6 group (−0.2 pp).

Limitations

  • Keep rate of only 18.7% for LLM-authored expressions implies high curation cost and potential selection bias toward cases GPT-4o finds easy to annotate.
  • Benchmark is static; because it is curated with GPT-4o assistance, future models trained on GPT-4o-generated data may exhibit a residual distributional advantage.
  • Evaluation of GPT-4o and Claude-3.5 relies on Set-of-Marks + Semantic-SAM, conflating segmenter quality with model reasoning quality.
  • Dataset size (5,000 instances) is smaller than RefCOCO/+/g (7,500+), limiting statistical power for fine-grained subgroup analyses.
  • The paper does not address REC in video or 3D settings, which are emerging evaluation frontiers.

Relevance to Vision-Language Models

Ref-Adv directly benchmarks where frontier MLLMs fail in fine-grained language-to-region grounding—a capability central to agent-based VLM pipelines, robotic manipulation, and document understanding. By exposing that >90% RefCOCO accuracy coexists with <65% accuracy on reasoning-demanding REC, the paper establishes that existing leaderboards overestimate grounding competence. The finding that CoT consistently helps on Ref-Adv but not on RefCOCO challenges standard evaluation practice and informs how future VLMs should be trained and assessed. The hard-distractor curation methodology is also directly applicable to constructing adversarial probes for other VLM benchmarks.