Skip to content

PoSh: Using Scene Graphs to Guide LLMs-as-a-Judge for Detailed Image Descriptions

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Scene graph-guided LLM-as-judge improves detailed image description evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PoSh is a reference-based evaluation metric for detailed image descriptions that extracts scene graphs from generated and reference texts, then uses an open-weight LLM-as-a-Judge with those graphs as structured rubrics to produce both granular (span-localized) and coarse (aggregate) scores for mistakes and omissions. It is paired with DOCENT, a new benchmark of 1,750 expert-described artworks with knowledgeable human judgments. PoSh outperforms all open-weight alternatives and GPT-4o-as-a-Judge in correlation with human ratings, and doubles as an effective RL reward signal.

Problem

Standard captioning metrics (CIDEr, SPICE, BLEU, CLIPScore) were designed for short texts and cannot reliably evaluate long, detailed image descriptions: they are insensitive to attribute/relation attachment errors (e.g., which agent is performing an action), they produce only a single coarse score with no error localization, and closed-model judges sacrifice replicability. Existing detailed-description benchmarks also lack granular human judgments, making it impossible to validate fine-grained metrics.

Method

PoSh operates in three steps on a (generation, reference) pair:

  1. Scene graph extraction — dependency parsing (en_core_web_trf) extracts sentence-level scene graphs \(G(d) = \langle O(d), E(d), K(d) \rangle\) (objects, attribute edges, relation edges), then co-reference resolution (maverick-mes-ontonotes) merges them into a single graph that preserves object attachment and localizes every component to source text spans.

  2. Granular scoring via QA — for each component \(c \in G(\text{gen})\) (and \(G(\text{ref})\)), templated presence questions are generated with disambiguating unique identifiers (resolved through a three-pass candidate search) and posed to qwen-3-14b, which predicts a 1–5 score; final scores are extracted as a weighted average over token logits. This yields per-component mistake scores \(\pi(c_\text{gen}) = \Psi(c_\text{gen}, \text{ref})\) and omission scores \(\rho(c_\text{ref}) = \Psi(c_\text{ref}, \text{gen})\).

  3. Coarse aggregation — \(\text{Mistakes} = \text{mean}_{c \in O(\text{gen})}(\pi(c))\) and \(\text{Omissions} = \text{mean}_{c \in O(\text{ref})}(\rho(c))\), averaging granular scores directly to maintain grounded interpretability.

DOCENT is constructed from 1,750 U.S. National Gallery of Art artworks with expert accessibility descriptions; 100 images receive generations from LLaVA-1.6-7B, Molmo-D-7B, GPT-4o, and Claude Sonnet 3.5, annotated by 24 art-history students with 300 granular and 600 coarse pairwise judgments.

Key Contributions

  • PoSh metric: open-weight, replicable, reference-based evaluation producing span-localized granular scores and aggregated coarse scores for mistakes (precision) and omissions (recall) in detailed image descriptions.
  • DOCENT benchmark: 1,750 expert-described artworks; 300 granular + 600 coarse human judgments from knowledgeable annotators; first benchmark with both judgment types for complex visual art.
  • Metric evaluation: PoSh achieves +0.05 Spearman \(\rho\) vs. best open-weight alternatives on DOCENT and +3 percentage points in pairwise ranking accuracy over GPT-4o-as-a-Judge; robustness confirmed on CapArena (web imagery).
  • PoSh as reward function: RL with DAPO using PoSh outperforms supervised fine-tuning (SFT) on DOCENT descriptions.
  • VLM characterization: even GPT-4o covers only 50.1% of visual information in DOCENT references, establishing a hard new benchmark for VLM progress.

Results

  • PoSh achieves +0.05 Spearman \(\rho\) with human judgments on DOCENT compared to the best open-weight alternatives (SPICE, CAPTURE, LLaVA-Critic).
  • PoSh recovers human description rankings +3 percentage points more often than existing metrics, surpassing GPT-4o-as-a-Judge.
  • PoSh is robust across image types, confirmed on CapArena (3,361 images, 14 VLMs, 5,599 human-judged pairs).
  • RL with PoSh as reward function outperforms SFT on DOCENT coarse pairwise judgments.
  • Best model (GPT-4o) achieves only 50.1% coverage of visual information in DOCENT references; all models struggle more with omissions than mistakes.
  • Student annotators show moderate inter-annotator agreement on coarse judgments (Krippendorff's \(\alpha = 0.509\), \(0.409\), \(0.459\) for mistakes, omissions, overall) with moderate-to-strong correlation with expert (Pearson \(\rho = 0.727\), \(0.501\), \(0.492\)).

Limitations

  • PoSh is text-only and reference-based: requires a gold reference description and cannot operate reference-free; unsuitable where ground-truth references do not exist.
  • Coarse scores are unweighted means over granular scores; component importance is treated as uniform, which may not match task-specific priorities.
  • Scene graph extraction relies on off-the-shelf dependency parsing and co-reference resolution, which can fail on complex or unusual sentence structures, propagating errors into scoring.
  • DOCENT annotators provide at least one judgment per task (only 15% get additional judgments), limiting statistical power for inter-annotator agreement estimates.
  • DOCENT covers a specialized domain (fine art); generalization of PoSh's calibration to other long-description domains (e.g., medical imaging, satellite imagery) is unvalidated.
  • The QA disambiguation multi-pass procedure is computationally heavier than single-pass metrics, though it remains cheaper than closed-API judges.

Relevance to Vision-Language Models

PoSh directly addresses a critical bottleneck in advancing VLMs beyond short-caption benchmarks: reliable, interpretable, and replicable evaluation of long-form image descriptions. For researchers tracking VLMs, it provides a practical tool to diagnose compositional failures — attribute misattachment, incorrect relational grounding — that coarser metrics conflate with broad quality. DOCENT fills a dataset gap by extending detailed-description evaluation to visually complex artwork with expert references, establishing a harder ceiling task where even frontier models (GPT-4o, Claude Sonnet 3.5) fall well short of full coverage. The demonstrated use of PoSh as an RL reward signal also connects directly to ongoing work on VLM post-training and self-improvement via verifiable reward functions.