S2H-DPO: Hardness-Aware Preference Optimization for Vision-Language Models

🕒 Published (v1): 2026-04-20 17:06 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Hardness-aware DPO closes multi-image reasoning gap in VLM alignment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

S2H-DPO introduces a Simple-to-Hard (S2H) preference optimization framework for VLMs that constructs multi-image DPO training data across three hierarchically ordered reasoning levels—single-image localized, multi-image localized, and global visual search. It addresses the gap in existing methods (notably MIA-DPO) that train only on index-specified, single-image-level queries and thus fail to build autonomous cross-image search and comparison skills. Training jointly on all three levels yields consistent gains on multi-image benchmarks without degrading single-image performance.

Problem

Existing multi-image preference alignment (e.g., MIA-DPO) trains exclusively on Level 1 queries that pre-specify which image to examine ("What is the color of the car in Image 2?"), sidestepping the harder skills of autonomous cross-image search and compositional reasoning. This leaves a capability gap: models cannot generalize to queries requiring global visual search or unprompted multi-image comparison. Additionally, MIA-DPO's chosen/rejected pair construction relies on model-specific hallucinations, requiring a new dataset per model architecture.

Method

S2H-DPO constructs 20K preference pairs per level across three hierarchically ordered task types and applies flat joint DPO training across all levels simultaneously.

• Level 1 (Single-Image Localized): Single-image VQA datasets are augmented with distractor images; rejected responses are hallucinated answers unrelated to the target image, following MIA-DPO.
• Level 2 (Multi-Image Localized): Two subtasks — Kinship Recognition (cross-image relational inference from face datasets) and Visual Arithmetic (counting/logical operators over synthetically generated shape images). Chosen/rejected pairs are deterministically derived from ground-truth labels.
• Level 3 (Global Visual Search): A target concept image from ImageNet is paired with \(N-1\) distractor images from different classes. The chosen response is a detailed caption of the target generated by Qwen2.5-VL-32B; the rejected response is generated by prompting the model to "caption the images" without specifying a target, yielding plausible-but-incorrect aggregations. Pairs with cosine similarity (CLIP/MPNet) above the top quartile threshold \(\tau\) are discarded to ensure contrastive signal quality.

Standard DPO loss is applied: $\(\mathcal{L}_{\text{DPO}}(\pi_\theta;\pi_{\text{ref}}) = -\mathbb{E}_{(x,y_w,y_l)\sim D}\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]\)$ with \(\beta=0.1\), trained for 3 epochs at \(5\times10^{-5}\) learning rate. Method is model-agnostic, tested on LLaVA-v1.5-7B, Qwen2.5-VL-7B, and Qwen3-VL-2B.

Key Contributions

• Identifies the capability gap in multi-image alignment: prior work covers only Level 1 (index-specified) reasoning, missing Levels 2–3.
• Proposes a prompt-driven complexity framework for generating chosen/rejected pairs that is model-agnostic (no dependence on architecture-specific hallucinations).
• Demonstrates that flat joint training across all three levels outperforms sequential curriculum training (L1→L2→L3), as sequential training causes myopic optimization biased toward localization cues.
• Shows DPO consistently outperforms SFT on the same multi-image data.

Results

Multi-image benchmarks (BLINK, MANTIS, NLVR2): - LLaVA-v1.5: +6.30%, +6.03%, +3.49% vs. base; average 43.7→48.97 (+5.27 pp); vs. MIA-DPO: +0.50, +2.34 (MANTIS), +1.39 (NLVR2) pp - Qwen2.5-VL-7B: +1.56, +5.53, +0.39 pp vs. base; average 65.74→68.24 (+2.49 pp); MIA-DPO degrades this model on BLINK (−13.01) and MANTIS (−9.21) - Qwen3-VL-2B: +2.31, +2.10, +0.90 pp; average 60.31→62.08 (+1.77 pp)

Single-image benchmarks (MMStar, POPE): - LLaVA-v1.5: MMStar 32.9→33.62, POPE 85.9→85.70 — no degradation - Qwen3-VL-2B: MMStar 53.42→53.62, POPE 85.28→85.46

Curriculum ablation (LLaVA-v1.5): Flat training on (L2∪L3) outperforms all sequential curricula; L1→L2→L3 performs worst (mean 45.55 vs. 48.97 for full flat S2H-DPO).

Limitations

• Level 2 tasks (Kinship Recognition, Visual Arithmetic) are narrow and may not represent the full diversity of real-world multi-image comparison tasks.
• Evaluation is limited to five benchmarks; BLINK and MANTIS may not fully capture all forms of global visual search needed in deployment.
• Rejected pair quality for Level 3 relies on Qwen2.5-VL-32B as a generator — quality is bounded by that model's captioning behavior.
• The semantic similarity filtering threshold \(\tau\) (top quartile) is a heuristic without ablation reported in the excerpt.
• 20K samples per level may not scale analysis to very large or diverse training regimes.

Relevance to Vision-Language Models

This work directly targets a known weak point of open-source VLMs relative to proprietary systems like GPT-4o: the ability to reason jointly across multiple images without being hand-held by index references. The S2H framework generalizes across architectures (LLaVA, Qwen-VL families) and avoids the fragility of hallucination-dependent data generation, making it practically deployable as a post-training alignment step. For researchers tracking VLMs, it establishes that hierarchical task decomposition in preference data construction — rather than scale alone — is a critical lever for multi-image capability alignment. It also reinforces the emerging consensus that DPO-style alignment on task-specific synthetic data is superior to SFT for nuanced visual reasoning.