Skip to content

MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization

🕒 Published (v1): 2026-07-16 17:02 UTC · Source: Arxiv · link

Why this paper was selected

Multimodal repo-level issue localization benchmark; exposes text-only gap in coding agent eval

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MM-IssueLoc is a controlled benchmark of 652 GitHub issue–PR instances (1,050 images, 23 languages) that isolates visual evidence as an explicit variable in repository-level issue localization. Existing systems reach at most 38.96 file Acc@5 and 33.86 function Acc@10, exposing a large capability gap not visible in text-dominant benchmarks. Strong scores on SWE-bench do not transfer to multimodal localization.

Problem

No existing benchmark isolates the contribution of visual evidence (screenshots, error dialogs, rendered UIs, logs) at the localization step. Text-only benchmarks strip images entirely; SWE-bench Multimodal retains images but evaluates end-to-end patch generation, so whether an image helped identify the correct file or function cannot be read off the final pass/fail signal. There are no per-image relevance annotations, paired with/without-image runs, or function-level gold labels purpose-built for this diagnostic.

Method

Each benchmark instance is a tuple \(x_i = (t_i, b_i, I_i, R_i@c_i)\) — issue title, body, image set, and repository snapshot at the PR base commit. Gold file sets \(F_i^\star\) are derived from PR diffs; gold function sets \(G_i^\star\) are extracted by intersecting unified-diff line ranges with tree-sitter AST spans (newly introduced functions excluded). Evaluation uses strict Acc@K:

\[\text{Acc@K} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[G_i^\star \subseteq \text{TopK}(r_i)]\]

requiring all gold locations to appear in the top-K ranked list.

Four paired input modes allow controlled isolation of visual evidence: text-only, with-image, +VCE (images replaced by structured textual fields extracted by a VLM: OCR text, error signals, UI elements, code hints, saliency), and +VCE+image. A controlled multimodal retriever, MM-IssueLoc-VL-Embedding, is trained via InfoNCE loss (\(\tau=0.05\)) on a disjoint 13,196-row training set with hard negatives mined from same-repository non-gold candidates. Training follows a two-stage curriculum: Stage 1 on file-level rows, Stage 2 on function-level rows initialized from Stage 1. A harmful-image subset (55 instances) is synthesized via three strategies: same-repository image-text mismatch, TF-IDF-similar issues with disjoint edit files, and promotion of misleading comment-thread screenshots.

Key Contributions

  • MM-IssueLoc benchmark: 652 issue-PR instances, 1,050 images, 23 languages, file-level gold for all instances, function-level gold for 343, with 7 image evidence categories and 4 relevance levels \(\{-1, 0, +1, +2\}\).
  • Controlled evaluation protocol with four input modes (text-only / with-image / VCE / VCE+image) enabling direct measurement of visual evidence contribution.
  • MM-IssueLoc-VL-Embedding: a multimodal retriever trained with curriculum contrastive learning on benchmark-disjoint data; achieves the best function-level retrieval result (Func@10 = 33.86).
  • Harmful-image robustness subset for testing whether systems are misled by visually plausible but localization-incorrect evidence.
  • Cross-benchmark analysis showing that high text-dominant SWE-bench scores (OpenHands: 94.53 File@5 on SWE-bench-Lite) do not transfer to multimodal localization (38.96 File@5 on MM-IssueLoc).

Results

  • Best file-level agent: OpenHands + GPT-5.2 → 38.96 File@5; best File@1/File@3 from OpenHands + Claude-Sonnet-4.6 (25.61 / 36.66).
  • Best function-level result: MM-IssueLoc-VL-Embedding-8B → 33.86 Func@10 (retriever), outperforming all LLM-based agents (best agent: 22.45 Func@10).
  • Difficulty collapse: OpenHands GPT-5.2 drops from 83.10 Acc@10 on easy instances to 2.84 on hard instances; MM-IssueLoc-VL-Emb-8B drops from 74.18 to 3.98.
  • Image ablation (\(\Delta_\text{img}\)): Removing images reduces MM-IssueLoc-VL-Emb-8B by 4.44 File@5 points and Emb-2B by 4.91 points; AgentLess GPT-5.2 loses 2.66 points; OpenHands agents change by less than 1 point, indicating inconsistent image exploitation by agents.
  • VCE effect on agents: model-dependent — Claude-Sonnet-4.6 drops 5.62 points, GPT-5.2 gains 4.74, Qwen3.5-122B-A10B drops 11.69 points.
  • Cross-benchmark gap: OpenHands reaches 94.53 File@5 on SWE-bench-Lite but only 38.96 on MM-IssueLoc; best SWE-bench-MM File@5 is 43.14.
  • Harness reliability effect: same Claude-Sonnet-4.6 backend gives OpenHands 37.42 File@5 (98.6% valid submissions) vs. Mini-SWE-Agent 18.71 File@5 (47.1% valid submissions).

Limitations

  • Function-level gold is available for only 343 of 652 instances (contingent on tree-sitter grammar coverage), leaving many languages evaluable only at file level.
  • Harmful images are synthetically constructed; naturally occurring harmful images are rare (<1% in human pool), so robustness results reflect controlled conditions rather than in-the-wild distribution.
  • Strict Acc@K penalizes partial retrieval equally regardless of how many gold locations are recovered; single-miss failures on multi-edit instances are not distinguished.
  • VCE is a fixed, non-adaptive image-to-text adapter; its benefit is model-dependent and the optimal structured fields may vary by evidence category.
  • AI-assisted annotation covers 202 instances (31%); although a four-stage VLM gate and a second-model consistency check are applied, annotation noise from this track is not fully characterized.
  • Benchmark instances are biased toward repositories with ≥50 stars, which may underrepresent lower-visibility codebases.

Relevance to Agentic AI / LLM Agents

MM-IssueLoc directly benchmarks a prerequisite step for software-engineering agents — repository-level issue localization — under realistic multimodal inputs that current agent evaluations (SWE-bench) have been treating as text-only. The finding that agent harness reliability (valid-submission rate) is a first-order performance factor highlights a critical engineering concern for deploying LLM agents on structured-output tasks. The controlled VCE ablations provide a reproducible diagnostic for testing whether an agent framework actively exploits visual evidence versus ignoring it, which is essential for building agents that handle the full modality of real developer workflows. For researchers developing multimodal tool-using agents or code-navigation agents (AgentLess, OpenHands, LocAgent-style systems), this benchmark fills the gap between end-to-end repair metrics and fine-grained localization capability assessment.