NL-Eye: Abductive NLI For Images

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NL-Eye is a benchmark that adapts the textual abductive NLI task to the visual domain, requiring VLMs to judge which of two hypothesis images is more plausible given a premise image and explain the choice. Across 350 carefully curated image triplets (1,050 images), modern VLMs largely fail at random-baseline levels while humans achieve 83–85% prediction accuracy and 95% explanation validity. The benchmark exposes a fundamental gap between VLMs' competent textual reasoning and their deficient visual interpretation.

Problem

Prior VLM benchmarks evaluate single-scene visual entailment or next-frame prediction but do not systematically probe cross-image abductive reasoning — the ability to infer the most plausible cause or outcome given a context scene. It is therefore unknown whether VLMs can handle real-world inference tasks such as warning about a wet floor or identifying the cause of a crying infant.

Method

NL-Eye frames visual abductive reasoning as a two-task benchmark: - Plausibility Prediction: given a premise image (IP) and one or two hypothesis images (IH), the model selects the more plausible hypothesis or assigns a 1–10 Likert plausibility score. - Plausibility Explanation: the model generates a free-text justification for its choice.

Data construction: Human annotators wrote high-quality textual scene descriptions; professional designers used Midjourney and DALL-E to generate photorealistic premise + two-hypothesis image triplets. Each triplet is annotated with (a) a gold plausibility label, (b) a gold explanation, (c) reasoning category (physical, logical, emotional, functional, social, cultural), and (d) temporal direction (forward/backward/parallel) and duration (short/long). Total: 350 triplets, 1,050 images.

Evaluation setups: - Triplet setup: model receives all three images and picks the more plausible hypothesis; consistency accuracy requires the model to be correct for both orderings of hypotheses. - Pairs setup: model scores each hypothesis individually; order-faithfulness accuracy checks that the plausible hypothesis receives a higher score.

Input strategies: (v.1) combined single image vs. (v.2) separate images; text-based ablations use (t.1) VLM-generated captions as text input and (t.2) gold textual descriptions.

Explanation evaluation: human AMT workers select valid explanations from a pool; automatic evaluation uses GPT-4o as a reference-based judge against gold + human-approved reference explanations.

Key Contributions

• Novel image-to-image abductive NLI benchmark with six reasoning categories and temporal annotations.
• Two complementary evaluation setups (triplet and pairs) and a consistency/order-faithfulness metric to control for order sensitivity.
• Comprehensive study across closed-source VLMs (GPT-4V, GPT-4o, Gemini-1.5-Pro, Claude-Sonnet-3.5, Claude-Opus-3) and open-source VLMs (LLaVA 1.6, Fuyu), plus text-only NLI baselines.
• Diagnostic finding that VLM failure stems from visual interpretation, not textual reasoning: text-only (gold description) experiments reach ~80% accuracy.
• First demonstration (to the authors' knowledge) of comparative vs. absolute judgment performance asymmetry in visual reasoning tasks.

Results

• Human performance: 85% triplet accuracy, 83% pairs order-faithfulness accuracy, 95% valid explanation rate.
• Best VLM (GPT-4o, combined image, triplet): 60% accuracy — the only model to exceed the dumb-pixel baseline (50%).
• All other VLMs in the separate-images triplet setup score 16–51%, with most at or below the random baseline (25%) or dumb-pixel baseline (50%).
• Open-source models (LLaVA 1.6: 14%, Fuyu: 4%) fall below random.
• Even when VLMs predict correctly, valid explanations are produced in at most ~50% of cases (vs. 95% for humans).
• Text-only (gold descriptions): GPTs and Claudes reach 78–81%, confirming textual reasoning competence.
• Image-to-text (VLM-generated captions fed to GPT-4o): drops to 29–40%, on par with vision-based failure — indicating a recognition/description gap.
• Separate images outperform combined images by >10% on average (48.6% vs. 37.3%), except GPT-4o.
• VLMs perform relatively better on Social and Cultural categories (knowledge-based); worst on Emotional and causal (forward/backward) reasoning; best on parallel (correlational) reasoning.

Limitations

• Small scale: 350 triplets limits statistical power and domain diversity.
• All images are synthetically generated (Midjourney/DALL-E), which may not reflect natural image distributions and introduces domain gap with real-world deployment.
• Gold labels and explanations reflect the cultural priors of a specific annotator pool, potentially amplifying bias.
• Evaluating only whether an explanation "aligns" with a reference explanation may penalize valid but differently worded reasoning.
• Benchmark evaluates current model snapshots; rapid VLM progress may reduce the observed gaps quickly.
• Text-to-image copyright status of the generated images is legally ambiguous.

Relevance to Vision-Language Models

NL-Eye directly probes a widely assumed but inadequately tested VLM capability — multi-image causal/abductive reasoning — and demonstrates that state-of-the-art models including GPT-4o, Gemini-1.5-Pro, and Claude-Sonnet-3.5 largely fail where non-expert humans succeed, exposing a fundamental gap in visual grounding rather than language understanding. The controlled text-only ablation cleanly localizes the bottleneck to the visual encoder/representation pathway, providing a concrete diagnostic for VLM architecture research. The benchmark also introduces the image-to-image entailment paradigm and consistency/order-faithfulness metrics that are reusable for evaluating future models. For VLM researchers, NL-Eye highlights that improving commonsense and causal visual reasoning — especially emotional and forward/backward temporal reasoning — is an open, high-priority challenge.