Skip to content

ImAgent: A Unified Multimodal Agent Framework for Test-Time Scalable Image Generation

🕒 Published (v1): 2025-11-14 17:00 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ImAgent is a training-free, single-model meta-harness for text-to-image generation that wraps multiple generation strategies—prompt enhancement, image refinement, best-of-N sampling—inside a unified multimodal model (UMM) and uses a policy controller to adaptively select and sequence them at inference time. Unlike prior pipelines that assemble separate models for reasoning, generation, and evaluation, ImAgent collapses all roles into one UMM backbone. It improves over its backbone on every benchmark tested without any fine-tuning.

Problem

Existing test-time scaling strategies for T2I generation (prompt rewriting, best-of-N sampling, iterative refinement) each require separate external models and operate as standalone modules. Composing them demands human-designed pipelines and manual strategy selection per input, leading to redundant computation and limited adaptivity. There is no unified, self-organizing framework that can allocate test-time compute automatically.

Method

ImAgent formalizes generation as a sequential decision process. At step \(t\), the state is \(s_t = \{P_0, I_0, P_t, I_t, \mathcal{O}_{t-1}\}\), where \(P_0, I_0\) are the initial prompt and image, \(P_t, I_t\) are the current versions, and \(\mathcal{O}_{t-1}\) is the observation history (action + prompt traces). A policy controller \(\pi_\theta\) selects an action \(a_t \sim \pi_\theta(a \mid s_t)\) from a fixed action space; the selected action \(f_{a_t}\) produces \((P_{t+1}, I_{t+1}, o_t)\) and the observation history is updated: \(\mathcal{O}_t \leftarrow \mathcal{O}_{t-1} \cup \{o_t\}\). The process runs until STOP is chosen or \(T_{\max}=5\) steps are exhausted.

The action space contains six operations: Naive Generation/Editing, Prompt Enhancement with CoT (enriches vague prompts via chain-of-thought in the language head), Prompt Revision (self-corrects the prompt based on visual feedback), Image Detail Refinement (generates a local editing sub-prompt to fix perceptual artifacts), Best-of-N Sampling (\(N=4\), scores candidates on semantic alignment), and STOP. Crucially, both the policy controller and all action executions run inside the same UMM; no external models are called. Actions are parsed via deterministic regex on \boxed{action_name} outputs, with a fallback to Naive Generation for any out-of-set prediction.

Key Contributions

  • A training-free agentic framework where policy control and all generation actions are co-located in a single UMM, eliminating inter-model communication overhead.
  • A structured decision prompt with observation history that enables the policy controller to select contextually appropriate actions without task-specific fine-tuning.
  • A regex-based action parser with fallback mechanism that robustifies agentic execution against malformed outputs.
  • Demonstrated test-time scaling across two backbone UMMs (Bagel, Janus-Pro-7B) and seven benchmarks covering generation and editing.

Results

  • WISE (image generation): +21.2% over Bagel vanilla; +25.7% over Janus-Pro-7B vanilla. ImAgent on Janus-Pro-7B surpasses SDXL, EMU3, and Show-o.
  • R2I-Bench: Bagel vanilla 0.54 → ImAgent 0.62 (+14.8%); Janus-Pro-7B vanilla 0.47 → ImAgent 0.51 (+7.5%).
  • T2I-ReasonBench: ImAgent on Bagel achieves overall quality score 89.0 vs. Gemini-2.0's 88.7 (surpasses the commercial model on quality); overall accuracy 51.4 vs. Gemini-2.0's 64.8.
  • RISEBench (image editing): Bagel vanilla 6.1 → ImAgent 13.1 (+114.8%); on par with Gemini-2.0 (13.3) and surpassing Qwen-Image-Edit (8.9) and Seedream-4.0 (10.8).
  • ImgEdit-Bench: Bagel vanilla 2.89 → ImAgent 3.15; best among open-source models, surpassing Step1X-Edit (3.06) and UniWorld-V1 (3.26 is higher—paper says ImAgent achieves "best performance among open-source models").
  • GEdit-Bench: +5.5% (English) and +5.2% (Chinese) over vanilla Bagel.

Limitations

  • ImAgent on Bagel underperforms the Self-Rewrite baseline on WISE; the paper attributes this to dataset-specific familiarity of Self-Rewrite, but the gap is a real benchmark liability.
  • Janus-Pro-7B does not support image editing, so the editing experiments use Bagel only—cross-backbone editing coverage is absent.
  • The maximum step cap (\(T_{\max}=5\)) is a hard limit on refinement depth; there is no dynamic budget allocation based on prompt complexity.
  • The policy controller is not fine-tuned for agentic decision-making, so its decisions depend entirely on the UMM's instruction-following quality; failure modes are papered over by a fallback rather than corrected.
  • No latency or wall-clock cost analysis is reported; test-time compute overhead is not quantified relative to baselines.

Relevance to Harnesses / Meta-Harnesses

ImAgent is a concrete instance of the single-model meta-harness pattern: rather than an external orchestrator wiring together specialist tools, the orchestration logic (policy controller) and all tool implementations (generation actions) share weights, reducing the harness to prompt engineering and a regex parser. This is a significant data point for harness designers—it shows that a UMM's understanding head can serve as a zero-shot meta-controller over its own generation head without any training, provided the action interface is kept simple and structured. The observation-history state formulation (\(s_t\)) and the fallback mechanism for invalid action outputs are directly portable patterns for any harness building on generative models that lack explicit agentic fine-tuning.