Semantic Browsing: Controllable Diversity for Image Generation¶

🕒 Published (v1): 2026-06-22 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Text-to-image models collapse semantically diverse prompts into near-identical outputs; Semantic Browsing fixes this by shifting diversity generation entirely to the text level via a structured multi-agent VLM workflow. The system builds a rooted hierarchical tree of JSON scene representations, where each edge encodes an explicit, traceable semantic constraint, then renders leaf nodes with any prompt-adherent T2I backend. Users can browse or interactively extend the resulting gallery of interpretable image variants.

Problem¶

Modern T2I models trained on highly detailed captions overfit to a single realization of an underspecified prompt; varying the random seed produces visually near-identical outputs (diversity collapse). Prior diversity methods—latent-space repulsion, conditioning perturbation, generate-and-filter—increase stochastic variation but provide no explicit user control over which semantic dimensions vary, making the results uninterpretable and unnavigable.

Method¶

The method operates entirely at the text/semantic level. A user prompt \(p\) is expanded by a VLM into a structured JSON scene representation \(s_0 \in \mathcal{S}\). A rooted tree \((V, E)\) is then grown iteratively: each edge \((s_1, s_2)\) encodes an atomic semantic constraint \(c\) that instantiates one aspect \(a\) (e.g., "Interactions → Lively play"), transforming the parent JSON via a VLM-based scene refiner \(R\) such that \(s_2 = R(s_1, c)\).

Tree growth is driven by a four-agent pipeline triggered per node: 1. Context Analyst — distinguishes prompt-locked details from VLM-filled, mutable details \(\{d_i\}\), enforcing plausibility. 2. Brainstormer — inductively aggregates low-level mutable details into high-level semantic aspects \(\{a_i\}\), rated by impact magnitude. 3. Decision Maker — selects the single highest-impact prompt-dependent aspect \(a^*\) and instantiates it into maximally divergent sibling constraints \(\{c_i\}\). 4. Critic — validates all candidates against the accumulated constraint trajectory \(\mathcal{C}_s = (c_1, \ldots, c_n)\), eliminates contradictions, and emits final instructions to a JSON Refiner.

Each branching uses 2 modification branches plus 1 identity ("preserve") branch. With branching factor 3 and depth 3, the gallery contains \(3^3 = 27\) leaf nodes. All agents reason over the full constraint history \(\mathcal{C}_s\) to maintain structural inheritance. Rendering is model-agnostic; experiments use FIBO as the VLM+prompt pipeline and optionally FLUX.2 as the T2I backend.

Key Contributions¶

Formal definition of Structured/Controlled Semantic Diversity as a new task distinct from stochastic diversity.
Semantic Browsing: a hierarchical tree formalism where each edge is a named, plausible, heterogeneous semantic constraint, enabling interpretable user navigation.
Multi-agent agentic workflow (Context Analyst → Brainstormer → Decision Maker → Critic → JSON Refiner) that enforces three tree requirements simultaneously: Semantic Structuring, Heterogeneity, Plausibility.
New dedicated evaluation metrics for hierarchical semantic consistency, beyond standard pairwise diversity scores.
Demonstration that the framework is model-agnostic (decoupled VLM reasoning from T2I rendering).
Support for interactive browsing: users can select any intermediate node and trigger further expansion.

Results¶

Outperforms all baselines on diversity metrics without degrading prompt alignment or image quality (reported as "consistent and substantial improvements").
Baselines compared: Stochastic VLM Seeding, Post-Hoc Diversity Optimization (79-candidate pool), High-Temperature Post-Hoc, CADS, Guidance Interval, Power-Law CFG.
VLM call budget matched to the stochastic baselines for fair comparison (79 total VLM calls for pool baselines).
Ablations confirm each of the four agents contributes to the final structured diversity (full ablation in paper).
Scaling ablation (Appendix F) explores varying tree depth and branching factor.
VLM-sensitivity study (Appendix E) shows robustness across VLM choices.
Works with FLUX.2 as an alternate renderer, demonstrating backbone-agnostic semantic control (Figure 9).

(Note: specific numerical scores—e.g., exact CLIP or FID values—are referenced but not reproduced in the provided excerpt.)

Limitations¶

Relies on prompt-adherent T2I backends (FIBO, FLUX.2); generators that ignore fine-grained textual differences would break the semantic→pixel mapping.
Tree depth and branching factor are fixed hyperparameters; automatic determination of optimal tree structure is not addressed.
Diversity is bounded by what the VLM can articulate as JSON-encodable semantic axes; purely visual or stylistic variation that lacks a textual description may be missed.
VLM-generated JSON representations may introduce hallucinated or inconsistent scene details not evident in the original prompt.
Interactive browsing requires user engagement; fully automatic gallery generation does not learn user preferences across sessions.

Relevance to Harnesses / Meta-Harnesses¶

This paper is a concrete instance of a structured multi-agent harness for controlled generation: the four-agent pipeline (Analyst → Brainstormer → Decision Maker → Critic) is a reusable orchestration pattern where each agent has a precisely scoped role, and their outputs chain sequentially with a shared context object (\(s\), \(p\), \(\mathcal{C}_s\)). The explicit JSON-based intermediate representation functions as a harness-level state contract between agents, analogous to how meta-harnesses pass structured artifacts between pipeline stages. The identity-branch mechanism (preserve) is a notable harness design choice that maintains intermediate states as valid renderable nodes—relevant to harnesses that need graceful partial-execution semantics. The work also illustrates how a meta-harness can decouple reasoning (VLM agents) from execution (T2I renderer), a modular architecture pattern directly applicable to general agent orchestration frameworks.