MIBE: Multi-subject Interaction Benchmark and Evaluator for Personalized Image Generation¶

🕒 Published (v1): 2026-07-01 18:44 UTC · Source: Arxiv · link

Why this paper was selected

Multi-subject personalized image generation benchmark; evaluates identity-binding in generative VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Existing automatic metrics (CLIP, DINO, reward models) fail to reliably evaluate multi-subject personalized image generation, collapsing toward random agreement as subject count increases. MIBE introduces a controlled benchmark (MIB) with 60K VLM-labeled and 4K human-labeled pairwise comparisons, and a learned evaluator (MIE) that achieves 0.922 pairwise accuracy against human preference, substantially outperforming all baselines. The key insight is that evaluation must jointly reason over three binding dimensions—Existence, Appearance, and Interaction—rather than aggregate into a single aesthetic score.

Problem¶

Multi-subject personalized image generation suffers from a "binding problem": models must render each reference identity correctly, preserve each subject's appearance without cross-subject leakage, and assign subjects to prompt-specified roles. Existing metrics are inadequate: CLIP/DINO-based scores lack instance-specific grounding, reward models (PickScore, HPS v2.1, ImageReward) prioritize aesthetics over binding, and no metric factorizes Existence/Appearance/Interaction simultaneously. On MIB-Gold, HPS v2.1 achieves only 0.520 pairwise agreement and PickScore falls to 0.486 (below random) as subject count grows toward 8.

Method¶

MIBE consists of two components:

MIB (benchmark): 15K hierarchical Level-8 seed prompts are decomposed into Level-6/4/2 variants by removing entity-relation pairs, yielding 60K prompt-reference tasks across 36 factorized buckets (4 subject counts × 3 human/object ratio types × 3 relation types: non-contact, occlusion, physical interaction). A fixed pool of 80 GPT-Image-1 reference subjects provides identity anchors. Candidate pairs are generated under matched prompt-reference conditions. - Silver Set (60K pairs): dual-VLM annotation using Gemini-2.5-Flash and Gemini-3.1-Flash-Lite under a structured SOP that decomposes each comparison into Existence, Appearance, and Interaction checks; pairs are retained only on consensus (95.1% agreement), yielding 56,909 usable pairs. - Gold Set (4,020 pairs): double-blind human annotation covering six generators (GPT-Image-1.5, Flux2, Seedream 4.5, GLM, Nano Banana, MOSAIC).

MIE (evaluator): A reference-conditioned multimodal evaluator maps \((R, p, y)\) to a scalar quality score \(s_\theta(R, p, y) \in \mathbb{R}\) and a diagnostic vector \(\hat{d}_\theta = (\hat{d}_{\text{exist}}, \hat{d}_{\text{app}}, \hat{d}_{\text{inter}})\). A dual-head architecture optimizes a joint loss \(\mathcal{L} = \alpha \mathcal{L}_{\text{rank}} + \beta \mathcal{L}_{\text{diag}}\), forcing the ranking signal to ground itself in concrete binding failures. Fine-tuning uses LoRA + layer updates over Qwen3.5-VLM (0.8B–4B) on only 9.6K of the 56.9K Silver pairs.

Key Contributions¶

MIB: A controlled, factorized benchmark decoupling subject count, composition type, and relation type, with separate silver (VLM-labeled) and gold (human-labeled) tiers to enable both scalable training and rigorous evaluation.
MIE: A lightweight, reference-conditioned evaluator with a dual-head ranking + diagnostic architecture trained entirely on MIB-Silver and validated against MIB-Gold.
Meta-evaluation findings: Systematic documentation that standard metrics collapse toward random agreement at higher subject counts, and that coupled Existence–Appearance failures and interaction-induced subject deformation are primary failure modes in state-of-the-art generators.

Results¶

MIE (qwen35_4b_lora_layer) achieves 0.922 overall pairwise accuracy vs. human preference on MIB-Gold; 0.982 on seen generators, 0.884 on unseen generators.
MIE reaches macro-F1 of 0.818 on the three diagnostic dimensions (Existence, Appearance, Interaction).
Baselines on MIB-Gold: PickScore = 0.4857 (below random), HPS v2.1 = 0.5201, PSNR = 0.3987 (actively misaligned); SCR, DINOv2, SigLIP-I show non-trivial but single-dimensional alignment.
LoRA-layer vs. layer-only at 2B: +0.061 pairwise accuracy, +0.116 macro-F1; at 4B: +0.046, +0.044.
Training on a 9.6K subset of Silver pairs is sufficient to achieve strong Gold alignment, demonstrating data efficiency.
Existence is the easiest diagnostic dimension; Appearance and Interaction remain the more discriminative challenge under generator distribution shift.

Limitations¶

The Gold Set's 80 reference subjects and white-studio setting cover a narrow visual distribution; long-tail identities and in-the-wild backgrounds are out of scope.
MIB-Gold is a temporal snapshot of six generators; newer architectures may introduce failure modes not represented in the current distribution.
Silver Set VLM labeling is susceptible to model-specific biases (e.g., conflating surface fidelity with binding accuracy) and may underperform on culturally underrepresented concepts.
The seen-to-unseen generalization gap (up to −0.221 for the weakest variant) confirms cross-generator generalization remains a non-trivial open problem.

Relevance to Vision-Language Models¶

MIE is itself a fine-tuned VLM (Qwen3.5-VLM with LoRA), demonstrating that lightweight VLM adaptation can serve as a human-aligned evaluator for complex compositional image generation tasks. The paper also critically characterizes VLMs-as-judges: the Silver Set's SOP-guided dual-VLM consensus pipeline (95.1% agreement) shows how VLMs can provide scalable pseudo-labels when prompted with structured decomposition rather than holistic ratings, directly informing best practices for VLM-based evaluation. For the broader VLM community, MIBE exposes a key failure mode of current VLM-conditioned generation: multi-subject binding in contact-rich and occlusion-heavy scenes, a regime where reference-conditioned understanding of identity and spatial relations remains unsolved.