Skip to content

Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ISG introduces a scene-graph-based evaluation framework for interleaved text-and-image generation, operating at four granularity levels (holistic, structural, block, image) with interpretable QA feedback. Paired with ISG-Bench (1,150 samples, 8 categories) and ISG-Agent (a Plan-Execute-Refine compositional baseline), the work reveals that current unified models severely underperform compositional frameworks, which themselves remain far below human performance on fine-grained criteria.

Problem

Existing benchmarks for interleaved text-and-image generation are inadequate: they focus on language-dominant tasks, lack reference answers making evaluation ambiguous, and rely on coarse MLLM-as-a-Judge holistic scoring that cannot capture structural compliance, cross-modal consistency, or fine-grained image-level accuracy.

Method

ISG parses a user query into a scene-graph-like structure where text and image blocks are nodes and their semantic relationships are edges. From this structure it derives evaluation questions at four levels: - Structural: LLM predicts expected block layout; direct structural matching against generated output. - Holistic: MLLM-as-a-Judge with golden reference and Analyze-then-Judge CoT scoring. - Block: Subject-object-relation tuples extracted per block; VQA module (GPT-4o) answers "Yes/No" or "1–10 score" questions about text↔image correspondence. - Image: Dependency-aware entity/attribute/relation tuples mapped to specific generated images; VQA verifies image content against ground-truth requirements.

Few-shot in-context learning and multimodal input are used in the question-generation module; VQA score (1–10) is preferred over Yes/No due to better human alignment. ISG-Agent operationalizes a compositional baseline via a Plan (GPT-4o, JSON tool plan) → Execute (tool library: SD3, UltraEdit, DynamiCrafter, SV3D, DreamMover) → Refine (Claude-3.5-Sonnet selector + self-correction on error logs) pipeline.

Key Contributions

  • ISG: First multi-granular, automatic evaluation framework for interleaved text-and-image generation using scene-graph structure; Pearson similarity with human judgments of 0.718 (block) and 0.907 (image).
  • ISG-Bench: 1,150-sample benchmark across 8 categories / 21 subcategories with vision-language dependency constraints and golden reference answers, prioritizing vision-centric tasks (style transfer, 3D transformation, image decomposition).
  • ISG-Agent: Plan-Execute-Refine compositional baseline achieving 122% holistic improvement over best unified model; structural accuracy 0.871 vs. prior best 0.385 (Gemini).
  • Empirical finding that MLLM-as-a-Judge suffers from "image-quality bias," awarding higher scores to visually appealing outputs regardless of instruction compliance.

Results

  • Unified vs. compositional (holistic): Best unified model (CoMM-MiniGPT-5) scores 2.961; best compositional (Claude+SD3) scores 6.254 — 111% improvement; ISG-Agent scores 6.262.
  • ISG-Agent structural accuracy: 0.871 vs. 0.385 (Gemini, prior best).
  • ISG-Agent block score: 5.515 avg vs. 5.515 for Gemini+SD3 (5.155 ablation without SOTA tools shows degradation); human baseline 7.611.
  • ISG-Agent image score: 0.574 avg vs. 0.150 (Gemini+SD2.1, best non-agent); human baseline 0.813.
  • Human alignment: Block-level Pearson 0.718 (score) vs. 0.446 (Yes/No); image-level accuracy 0.907; holistic agreement 0.730 with golden answer vs. 0.537 without.
  • Refinement ablation: Removing refinement drops block score from 5.515 to 5.366; removing SOTA tools drops it to 5.155.
  • All unified models produce 0–3 images in multi-image tasks and cannot be evaluated at block/image levels; Show-o exhibits hallucination (images follow system prompts rather than user instructions).

Limitations

  • ISG-Agent relies on GPT-4o for planning/verification and Claude-3.5-Sonnet for tool selection, making the pipeline expensive and dependent on proprietary models.
  • Compositional frameworks inherently cannot perform end-to-end image editing tasks (e.g., accurate style transfer) because perception and generation are decoupled.
  • ISG-Bench covers 1,150 samples; coverage of all interleaved generation scenarios remains incomplete (e.g., multi-turn, open-domain).
  • CLIPScore fails at block level because generated text blocks exceed the 77-token encoder limit, forcing reliance on MLLM VQA.
  • MLLM-as-a-Judge (holistic level) still shows image-quality bias even when provided with golden answers; holistic scores do not reliably reflect fine-grained instruction compliance.
  • Vision-dominated tasks systematically underperform across all models, suggesting current training data lacks sufficient vision-dominant interleaved instruction tuning.

Relevance to Vision-Language Models

ISG directly addresses the evaluation gap for the emerging class of unified VLMs (Show-o, Anole, Transfusion) that aim to both understand and generate images, exposing that they lag far behind compositional LLM+diffusion pipelines on structured multimodal tasks. The scene-graph evaluation paradigm extends TIFA-style VQA assessment to cross-modal consistency checking, which is critical for benchmarking next-generation VLMs that must maintain coherence across multiple generated text-image pairs. The identified "image-quality bias" in MLLM-as-a-Judge is a methodological warning relevant to any VLM evaluation relying on GPT-4o-style holistic scoring. ISG-Bench's vision-centric task taxonomy (style transfer, 3D transformation, image decomposition) defines a harder test bed for VLMs than prior language-dominant benchmarks.