Skip to content

CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Models

🕒 Published (v1): 2026-07-14 14:01 UTC · Source: Arxiv · link

Why this paper was selected

CoRe framework for cross-image comparative reasoning; fine-grained attribute grounding; fills known VLM compositional gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CoRe introduces a unified framework for cross-image comparative reasoning in VLMs, a capability where models must compare fine-grained metric attributes (count, depth, distance, spatial relations) across multiple images rather than analyzing each independently. It provides a 20K-sample training dataset, a new benchmark, and a structured reward framework (TriSR) that supervises intermediate reasoning steps under GRPO optimization. The approach yields a 28.2-point partial accuracy gain over the strongest baseline on the new CoRe-Bench.

Problem

Existing VLMs fail systematically at cross-image comparative reasoning—tasks requiring precise comparison of metric attributes across related images. Current benchmarks either target single-image perception or holistic multi-image semantics (temporal ordering, retrieval), leaving fine-grained comparative reasoning unevaluated. Outcome-only RL training exacerbates the problem: up to 17.1% of correct-answer predictions are accompanied by factually invalid reasoning chains, and LLM-as-Judge evaluation misaligns with human judgment on reasoning quality in this setting.

Method

CoRe has three integrated components:

CoRe-20K (training data): A metadata-driven, fully automatic triplet construction pipeline with three expert modules: (1) a Metric Extraction Expert that derives per-image numeric or categorical attributes from existing ground-truth annotations (crowd counts, depth maps, 3D bounding boxes); (2) a Quality Control Expert that filters triplets via a noise-margin lower bound \(\theta^m_{\min}\) (excluding near-equal pairs) and an upper bound \(\theta^m_{\max}\) (excluding trivially easy pairs), retaining only triplets where \(\delta_{ij} \in \mathcal{T}_m\) for all pairs; and (3) a Question Generation Expert that instantiates multiple-choice comparison questions from templates with option randomization to prevent position bias. Each triplet \((I_1, I_2, I_3)\) yields three coupled pairwise sub-questions.

TriSR (structured reward): Three verifiable reward signals combined with task accuracy under GRPO: - Attribute Grounding \(R_{\text{ag}}\): measures per-image attribute estimation accuracy via \(s_m(\hat{v}_k, v_k) = \exp(-|\hat{v}_k - v_k| / \max(|v_k|, \epsilon))\), averaged over three images. - Judgment Alignment \(R_{\text{ja}}\): checks that intermediate pairwise comparisons in the reasoning trace match both ground truth and the model's own final answer. - Triplet Consistency \(R_{\text{tc}}\): enforces transitivity—if \(I_1 > I_2\) and \(I_2 > I_3\), the predicted relation between \(I_1\) and \(I_3\) must be compatible; for interval answers, checks \(\Delta_{12} + \Delta_{23}\) is compatible with \(\Delta_{13}\).

Composite reward: \(R(o) = R_{\text{task}}(o) + \lambda_{\text{ag}} R_{\text{ag}}(o) + \lambda_{\text{ja}} R_{\text{ja}}(o) + \lambda_{\text{tc}} R_{\text{tc}}(o)\) with \(\lambda_{\text{ag}}=0.2\), \(\lambda_{\text{ja}}=0.2\), \(\lambda_{\text{tc}}=0.3\). GRPO computes group-normalized advantages \(\hat{A}_i = (R(o_i) - \text{mean})/\text{std}\) over \(G\) sampled rollouts per query.

Base model: Qwen3-VL-4B-Thinking, fine-tuned for 1 epoch on 8Ă— A800 GPUs.

Key Contributions

  • Identification of fine-grained cross-image comparative reasoning as a distinct, under-served VLM capability.
  • CoRe-20K: 20,000 triplets (16K train / 4K eval) spanning counting, depth, distance, and spatial relations, constructed automatically from structured metadata with deterministic ground-truth labels—no VLM pseudo-labeling.
  • TriSR: A structured reward framework that decomposes holistic reasoning quality into three verifiable, executable criteria (Attribute Grounding, Judgment Alignment, Triplet Consistency) and integrates them with GRPO.
  • CoRe-Bench: The first benchmark dedicated to fine-grained cross-image comparative reasoning (4,000 held-out triplets, 4 dimensions).

Results

  • CoRe achieves a +28.2-point gain in partial accuracy over the strongest VLM baseline on CoRe-Bench.
  • Qualitative failure analysis shows that frontier models—Qwen3-VL, GPT-5.2, Claude-Sonnet-4.5, Gemini-3-Pro, GPT-4o—all fail systematically on counting, depth, distance, and spatial relation tasks in the cross-image setting.
  • Diagnostic analysis (Table 4): outcome-only optimization leaves 17.1% of correct-answer responses paired with factually invalid reasoning chains; TriSR reduces this rate.
  • CoRe remains competitive on broader multimodal benchmarks (no significant regression on standard VLM evaluations).

(Note: full numeric tables beyond the 28.2-point headline and the 17.1% reasoning-error rate are not included in the provided text.)

Limitations

  • Base model is Qwen3-VL-4B-Thinking only; generalization across architectures and scales is not demonstrated in the provided text.
  • CoRe-20K is restricted to four metric dimensions (counting, depth, distance, spatial relations); other comparative attributes (e.g., texture, lighting, semantic similarity) are not covered.
  • The metadata-driven construction relies on availability of structured ground-truth annotations in source datasets, limiting applicability to datasets lacking such annotation.
  • Triplet formulation is fixed at three images; extension to \(k\)-way comparisons is not addressed.
  • Evaluation against the LLM-as-Judge baseline relies on a single judge model (Qwen3-VL-32B-Instruct), which may not generalize the alignment-gap finding.

Relevance to Vision-Language Models

CoRe directly targets a systematic gap in current VLMs: the inability to ground and compare metric attributes across multiple images with logically consistent reasoning. By showing that even GPT-5.2, Gemini-3-Pro, and Claude-Sonnet-4.5 fail on these tasks, the paper establishes cross-image comparative reasoning as a concrete frontier for VLM capability research. The TriSR reward design is broadly transferable—decomposing holistic reasoning correctness into verifiable intermediate signals is a principled alternative to LLM-as-Judge for any VLM task where intermediate steps have checkable structure. The metadata-driven dataset construction strategy also provides a scalable template for building high-quality multi-image training data without relying on VLM-generated pseudo-labels, which matters for any researcher studying multi-image or compositional visual reasoning.