Skip to content

Search Beyond What Can Be Taught: Evolving the Knowledge Boundary in Agentic Visual Generation

🕒 Published (v1): 2026-07-09 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Agentic visual generation with evolving knowledge boundary; directly applicable to VLM agent building

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Visual generators confidently hallucinate what they do not know, failing catastrophically (21–28/100) on prompts requiring world knowledge beyond their training cutoff. SearchGen introduces a noise-resistant gate–filter–integrate agentic search protocol and a co-training framework that jointly calibrates an 8B search reasoner with the visual generator, teaching the reasoner to search only for knowledge the generator cannot internalize. A single co-training iteration produces monotonic gains exceeding frontier open generators paired with frontier VLMs.

Problem

Visual generators trained on fixed corpora have a structural world-knowledge bottleneck: user requests are unbounded, evolving, and deeply long-tailed (new entities, post-cutoff events, niche cultural symbols), yet generators always produce an image with no mechanism to recognize ignorance. Naive search augmentation fails because it retrieves indiscriminately—injecting noise for prompts the generator already handles—and the critical divide between what knowledge can be parameterized vs. must remain in external context (the knowledge boundary) is generator-specific and shifts as the generator improves.

Method

Dataset construction. 20,840 production prompts are analyzed to distill twelve failure categories, yielding SearchGen-20K (20,839 bilingual prompts, 22 domains). Each prompt is annotated with fine-grained visual checklists and knowledge-gap slots (mean 5.2 per prompt). SearchGen-Corpus-1M pre-executes and freezes 145,642 image/web search sessions to enable fully offline, reproducible research.

Agentic reasoner (gate–filter–integrate). A three-stage pipeline controls tool use per knowledge gap: (1) a gate decides whether to search (NoSearch vs. VisualSearch vs. TextualSearch), (2) a filter discards noisy or irrelevant retrieved content, and (3) an integrate stage organizes retained evidence into multimodal generator context.

Co-training framework (teach-then-search). Two interlocking training loops discover the knowledge boundary empirically: - Online iterative DPO trains the generator to internalize knowledge it can absorb and to build noise resistance to imperfect search-augmented inputs. - Rejection-sampling fine-tuning (RSFT) calibrates the search reasoner to trigger search only for prompts the co-trained generator still cannot handle.

This creates a recursive self-improvement flywheel: the generator expands its parametric knowledge, shifting the boundary, and the reasoner adjusts its search scope accordingly.

Key Contributions

  • SearchGen-20K / SearchGen-Bench: 20,839 world-knowledge-grounded prompts across 12 failure categories and 22 domains with automated nine-component evaluation (knowledge-sensitive + rendering-quality components).
  • SearchGen-Corpus-1M: Pre-executed, frozen search harness (559,973 unique URLs, 370,733 cached downloads) enabling fully offline preference learning and search-policy research without live API keys.
  • 40-point evaluation gap: Demonstrated that frontier open generators collapse by up to 40 points on search-intensive prompts, a gap invisible to existing benchmarks.
  • Knowledge boundary analysis: Formalized the generator-specific, evolving boundary between parameterizable and contextual knowledge, and showed naive blind search degrades a substantial fraction of prompts.
  • Co-training recipe: Minimal 8B agentic reasoner + generator co-training via DPO + RSFT that outperforms pure generation and undiscriminating search.

Results

  • Frontier open-weight generators (Bagel, Flux.2-Klein-4B/9B, Qwen-Image) score 18–25/100 on knowledge-sensitive components (Checklist, Rubric, Visual Reference) on Search-Intensive prompts vs. 63–75/100 on NoSearch prompts — a collapse of up to 39.1 points.
  • Commercial systems with integrated search (GPT-Image-2: 71.2/100 overall; Nano Banana Pro: 64.4/100) confirm that reasoned search maintains performance; GPT-Image-2 drops only 0.1 points across strata.
  • Table 5 (partially visible) shows that blind search degrades VisualSearch stratum and TextualSearch stratum scores relative to no-search baseline, while the generator-adaptive search produces monotonic gains across all three strata (NoSearch → VisualSearch → TextualSearch).
  • Single-iteration co-training exceeds frontier open generators paired with frontier VLMs on SearchGen-Bench.

Limitations

  • The co-training recipe is described as deliberately minimal (single iteration); full recursive self-improvement cycles are proposed but not yet validated in the paper.
  • Knowledge boundary is generator-specific and shifting — a search policy calibrated to one generator version may harm a stronger future version without re-calibration.
  • Relies on Google Search API (image + web) for live deployment; offline reproducibility depends on the pre-frozen corpus, which will drift from the live web over time.
  • Dataset is bilingual (English/Chinese only), limiting direct generalization to other language communities despite covering 22 domains.
  • Evaluation relies on VLM-based automated judges; inter-rater agreement and judge accuracy are not fully characterized in the visible text.

Relevance to Agentic AI / LLM Agents

This paper directly addresses a core agentic decision problem — when to call a tool — in the domain of visual generation, operationalizing the concept of a generator-specific knowledge boundary to determine search invocation. The gate–filter–integrate pipeline and teach-then-search co-training framework are generalizable templates for calibrating any tool-augmented agent to avoid both over-calling (noise injection) and under-calling (hallucination). The recursive self-improvement framing — where the generator's expanding parametric knowledge continuously reshapes the search agent's policy — is a concrete instantiation of the broader agentic RL agenda of agents that improve their tool-use strategies through experience. The frozen SearchGen-Corpus-1M also provides a rare offline substrate for reward modeling and search-policy research without live API dependencies.