Skip to content

MMReD: a Cross-Modal Benchmark for Dense Context Reasoning

πŸ•’ Published (v1): 2026-01-01 Β· Source: ICLR Β· Venue: ICLR 2026 Β· link

Why this paper was selected

ICLR 2026; MMReD: dense multi-modal context reasoning over long inputs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

πŸ’¬ Ask ChatGPT✦ Ask Claude

TL;DR

MMReD is a cross-modal benchmark (24 tasks, up to 128 observations) that measures dense context reasoning β€” where every frame is informative and must be integrated globally β€” rather than sparse needle retrieval. All tested LLMs and LVLMs (β‰ˆ30 models) show monotonic accuracy degradation as context length grows, with leading reasoning-specialized models reaching 0% on certain tasks at 128 observations. Standard fine-tuning (SFT, GRPO) fails to generalize to longer contexts.

Problem

Existing long-context benchmarks (RULER, BABILong, Michelangelo, LongBench v2, VideoMME) are dominated by Needle-in-a-Haystack (NIAH) formulations, where only a sparse subset of context frames is relevant and the rest is distractor content. These benchmarks do not assess whether models can uniformly integrate information across all context elements. NIAH performance has no clear correlation with performance on densely informative reasoning tasks, leaving this capability unmeasured.

Method

The benchmark generates synthetic sequences of length \(N \in \{1, 2, 4, 8, 16, 32, 64, 128\}\) depicting 5 characters moving across 6 rooms. Each sequence state is rendered as a 512Γ—512 image (rooms as a 2Γ—3 grid of rectangles, characters as colored circles). LVLMs receive the sequence as images; LLMs receive a JSON text representation. 24 question types are divided into two categories:

  • NIAH questions (15 types): answerable from a single frame β€” First Appearance, Final Appearance, and Frame-X subgroups covering object tracking, counting, and spatial reasoning.
  • Dense Context (DC) questions (9 types): require uniform attention across all frames (e.g., "Which room was empty for the most steps?", "How many times did a crowd appear?").

Each sequence is paired with exactly one question; answers are algorithmically precomputed. The dataset contains 9,600 \((sequence, question, answer)\) triplets evaluated by exact-match accuracy. Fine-tuning experiments used SFT (Qwen2.5-7B-Instruct, Falcon3-Mamba-7B-Instruct) and GRPO (DeepSeek-R1-Distill-Qwen-7B) on \(N \in [1,2,4,8,16]\), testing generalization to \(N \in \{32, 64, 128\}\).

Key Contributions

  • MMReD benchmark with 24 tasks explicitly targeting dense (non-retrieval) multi-modal reasoning, scalable beyond 128 frames.
  • Empirical demonstration that all tested state-of-the-art LLMs, LVLMs, and reasoning-specialized models fail progressively on dense context tasks, reaching 0% accuracy on some DC tasks at \(N=128\).
  • Evidence that multimodal instruction tuning degrades long-context understanding relative to text-only counterparts.
  • Demonstration that SFT and GRPO fail to generalize dense context reasoning to unseen longer sequence lengths.
  • Theoretical grounding: results align with proofs that softmax attention disperses over irrelevant keys at scale (VeličkoviΔ‡ et al., 2025) and that transformers learn length-specific solutions with poor cross-length weight sharing (Ebrahimi et al., 2026).

Results

  • Universal degradation: All models degrade significantly for \(N > 32\); DC tasks degrade faster than NIAH tasks.
  • Best LLM (text): GPT-4o ('2024-11-20') achieves ~95% at \(N=1\), dropping to ~53% at \(N=128\) on average across all tasks; on DC tasks the drop is sharper.
  • Best LVLM (image): Qwen2.5-VL-72B-Instruct starts at ~95% (\(N=1\)), falls to ~38% at \(N=128\).
  • Reasoning models: DeepSeek-R1-Distill-Qwen-32B and Qwen3-32B-think score ~96–97% at \(N=1\) but reach 0% accuracy on certain DC tasks at \(N=128\) (8% for DeepSeekR1-Qwen-1.5B at \(N=128\), 58% for DeepSeekR1-Llama-70B).
  • LVLM vs. LLM gap: GPT-4o text outperforms GPT-4o image at all context lengths; InternVL2.5 performance degrades from \(N=16\) images despite a claimed 64-image capacity.
  • Fine-tuning failure: SFT (Qwen2.5-7B) and GRPO (DeepSeekR1-Qwen-7B) and Falcon3-Mamba-7B-SFT do not generalize; fine-tuned models at \(N=128\) score 53%, 27%, 28% respectively β€” worse than or comparable to the non-fine-tuned GPT-4o.
  • Coding models: Qwen2.5-Coder underperforms its general-purpose Qwen2.5 counterpart, indicating coding fine-tuning does not transfer to logical chain reasoning.

Limitations

  • Synthetic, minimalist visual environment (colored circles in grid rooms) may not reflect real-world visual complexity; tasks assume basic OCR but avoid perceptual challenges by design.
  • Evaluation is restricted to exact-match accuracy on a closed set of answer types (room names, character names, integers), precluding open-ended or free-form assessment.
  • Text representation for LLMs (JSON) is qualitatively different from the image representation for LVLMs, making cross-modal comparisons partially confounded.
  • Benchmark currently caps at 128 observations; though extensible, reported results are bounded by this length.
  • Fine-tuning experiments are limited to 7B-class models, and GRPO is tested only on one model; generalization failure may be scale-dependent.

Relevance to Vision-Language Models

MMReD directly exposes a structural weakness in current LVLMs: multimodal instruction tuning appears to impair long-context integration, as every tested LVLM underperforms its text-only LLM counterpart at equivalent context lengths. This matters for VLM research because dominant evaluation paradigms (VideoMME, MuirBench, Visual Haystacks) rely on sparse-frame retrievability, systematically missing the failure mode MMReD reveals. The benchmark provides a controlled diagnostic for studying how visual token compression schemes (e.g., 256 tokens/image in InternVL2.5) interact with context length limits, and it grounds observed failures in theoretical results about softmax attention dispersion β€” suggesting that architectural changes beyond context-window extension are needed for truly capable multimodal long-context reasoning.