Skip to content

ExpertVerse: A General-Purpose Benchmark for Expert-Level Reasoning in Knowledge-Intensive Visual Synthesis

🕒 Published (v1): 2026-07-21 17:59 UTC · Source: Arxiv · link

Here is the digest. I'll output the Markdown body as requested, and then write it to disk.

Why this paper was selected

Expert-level visual reasoning benchmark; stress-tests knowledge-intensive VLM synthesis beyond commonsense

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ExpertVerse is a knowledge-intensive visual synthesis benchmark of 1,611 expert-annotated instances stratified across 9 cognitive capabilities × 8 expert disciplines = 58 sub-tasks, covering single-image editing, multi-image composition, and text-to-image generation. The authors scale it into ExpertVerse-100K (with CoT traces and knowledge-anchored rationale annotations) and train KnowThinker—a VLM reasoner fine-tuned via SFT and a novel Bootstrapped Pareto Policy Optimization (BPPO) RL algorithm—achieving 69.73 average score versus 64.22 for SFT-only and approaching proprietary GPT-Image-2 (79.51).

Problem

Existing reasoning-aware image generation benchmarks (RISEBench, KRISBench, UniREditBench) focus on explicit commonsense reasoning, shallow causality, and direct knowledge recall, and lack a universal capability-centric taxonomy. They are also largely restricted to single-image manipulation, leaving knowledge-intensive multi-hop expert-domain reasoning across diverse generation types (multi-image composition, T2I synthesis) unexamined.

Method

Benchmark construction. Tasks are organized along an orthogonal 9×8 taxonomy: 9 cognitive capabilities (knowledge application, creative design, logical reasoning, aesthetic judgment, spatial transformation, temporal reasoning, narrative construction, cross-domain transfer, long text comprehension) × 8 expert categories (science & education, fashion & beauty, IP & entertainment, data & information, geography & history, product & design, film & storytelling, lifestyle & consumer) = 58 sub-categories. Handcrafted seed prompts are scaled via a proprietary VLM using a hierarchical expansion strategy; dual-stage QA (automated VLM filter + manual PhD verification) enforces factual correctness. The 1,611 instances span single-image editing (66.22%), T2I synthesis (26.17%), and multi-image composition (7.5%), with Chinese/English bilingual annotations. ExpertVerse-100K is generated from the same pipeline with step-wise CoT traces and knowledge-anchored rationale annotations; chart-heavy instances use an HTML-based rendering pipeline.

KnowThinker training (thinker-centric paradigm). The VLM thinker \(T\) is isolated from a frozen visual editor \(G\), producing reasoning trace \(R = \{r_1, \ldots, r_k\}\) and refined instruction \(q'\) via \((R, q') = T(p, I_\text{ref}; \phi)\), then \(I_\text{out} = G(q', I_\text{ref}, R)\). Training: (1) SFT on ExpertVerse-100K via LoRA (\(r=16\), \(\alpha=32\)) to bootstrap world-knowledge distillation; (2) RL with BPPO using difficulty-aware distillation that retains instances where group reward mean \(\mu \in [\tau_\text{low}, \tau_\text{high}]\) and variance \(\sigma^2 \geq \tau_\text{var}\).

BPPO algorithm. - BRR (Bootstrapping Reward Rectification): addresses cross-modal credit misalignment by calibrating the GRPO baseline with a cross-modal anchor \(s^{\bar{m}}_i\). Rectified baseline: \(\tilde{b}^m_i = (1-\beta)\mu^m + \beta s^{\bar{m}}_i\). Calibrated advantage: \(\tilde{A}^m_i = A^m_i - \beta(s^{\bar{m}}_i - \mu^{\bar{m}})/\sigma^m\). This penalizes rollouts that excel in one modality while failing in the paired modality. - CPAF (Conflict-Aware Pareto Advantage Fusion): detects sign conflicts between \(A^\text{img}_i\) and \(A^\text{cot}_i\). When signs agree, linear fusion with weight \(\alpha\); when they conflict: \(C(A^\text{img}_i, A^\text{cot}_i) = \arg\min_{A \in \{A^\text{img}_i, A^\text{cot}_i\}} |A|\), retreating to the weaker signal to preserve Pareto optimality.

Evaluation. Four dimensions scored 1–5: Knowledge Reasoning, Visual Quality, Input Consistency, Rationale Alignment; assessed by a proprietary multimodal evaluator with dimension-specific prompts.

Key Contributions

  • ExpertVerse benchmark: 1,611 expert-annotated instances across 58 sub-tasks (9 capabilities × 8 disciplines), first to jointly cover S-I2I, M-I2I, and T2I with bilingual annotations and expert-level knowledge density
  • ExpertVerse-100K: large-scale training corpus with step-wise CoT traces, knowledge-anchored rationale annotations, and an HTML-based rendering pipeline for data-visualization instances
  • KnowThinker: thinker-centric VLM reasoner (SFT + BPPO RL) producing explicit reasoning traces and refined instructions without jointly optimizing the visual editor
  • BPPO: BRR resolves cross-modal credit misalignment at the baseline level; CPAF resolves multi-objective gradient conflicts in the advantage space via sign-based conflict detection

Results

From Table 2 (average score across 8 knowledge domains): - KnowThinker with BPPO (ours): 69.73 avg (open-source, I2I+T2I) - KnowThinker SFT only: 64.22 - GPT-Image-2 (proprietary): 79.51 - NanoBanana Pro (proprietary): 68.89 - Seedream-4.0 (proprietary): 64.50 - NanoBanana (proprietary): 46.33 - GPT-Image-1 (proprietary): 41.52 - Qwen-Image (Edit): 38.81 - DreamOmni2: 29.28 - Uni-CoT: 26.45 - BAGEL (w/o CoT): 20.31 - Step1X-Edit: 16.23

From Table 3 (partial, reasoning sub-dimensions for community models): - Unified Thinker: 61.9 reasoning; ThinkRL-Edit: 61.7 reasoning - KnowThinker rows in Table 3 are truncated in the source text

Limitations

  • Visual editor is frozen throughout training; joint thinker–generator co-optimization is unexplored and may yield further gains
  • ExpertVerse-100K annotations are generated by a proprietary VLM, introducing an oracle ceiling and limiting full pipeline reproducibility
  • Evaluation relies on a proprietary multimodal evaluator; public score replication requires the same closed model
  • Multi-image composition represents only 7.5% of benchmark instances; T2I 26.17%—single-image editing dominates
  • BPPO hyperparameters (\(\beta\), \(\alpha\), distillation thresholds \(\tau_\text{low}\), \(\tau_\text{high}\), \(\tau_\text{var}\)) require non-trivial tuning

Relevance to Vision-Language Models

ExpertVerse operationalizes VLMs as knowledge-intensive reasoning engines that externalize world knowledge into step-wise visual plans, probing capabilities (multi-hop expert reasoning, cross-domain transfer, long-text comprehension) that standard VLM benchmarks do not stress-test. The benchmark's four-dimensional evaluation protocol—especially Knowledge Reasoning and Rationale Alignment—offers a more rigorous lens for assessing VLM grounding in professional knowledge than perceptual fidelity metrics alone. BPPO's treatment of cross-modal credit misalignment and gradient conflicts in multi-reward RL is broadly applicable to any VLM trained with joint text-and-image reward signals. The thinker-centric paradigm also provides a clean framework for isolating VLM reasoning quality from downstream generator quality in modular multimodal pipelines.