Skip to content

GenIR: Generative Visual Feedback for Mental Image Retrieval

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GenIR introduces Mental Image Retrieval (MIR), a multi-round interactive retrieval task where users iteratively refine text queries to find a mentally envisioned target image. Instead of verbal feedback, GenIR generates a synthetic image from each query via a diffusion model and uses it for image-to-image retrieval, making the system's "visual belief" explicit and actionable. This generative visual feedback paradigm substantially outperforms all verbal and prediction-feedback baselines across four diverse benchmarks.

Problem

Existing multi-round interactive text-to-image retrieval systems (e.g., ChatIR, PlugIR) rely solely on verbal (text-based) feedback, which is ambiguous, potentially misleading, and fails to expose the system's internal vision-language representation. Minor textual edits in CLIP-like embedding spaces produce unpredictable retrieval changes, making query refinement a blind trial-and-error process. No prior work had formally defined or benchmarked the "known-item search with mental image" scenario as a distinct task.

Method

GenIR implements a closed-loop, round-by-round pipeline: 1. Query formulation: the user (or a VLM proxy, Gemma3) writes a text query q_t describing their mental image. 2. Synthetic image generation: a text-to-image diffusion model G produces I_t^synthetic = G(q_t), externalizing the system's current query interpretation into image space. 3. Image-to-image retrieval: both the synthetic image and all database images are encoded with CLIP's image encoder; the database image maximizing cosine similarity to I_t^synthetic is returned. 4. Feedback loop: the user observes I_t^synthetic, identifies discrepancies from their mental target, and rewrites q_{t+1}.

The framework is model-agnostic—any text-to-image generator (Infinity, Lumina-Image-2.0, SD3.5, FLUX.1, HiDream-I1) and any image encoder can be plugged in. An automated dataset curation pipeline runs the same loop with a VLM replacing the human, producing tuples (q_t, I_t^synthetic, I_t^retrieved, y_t) as training/evaluation data.

Key Contributions

  • Task definition: Formal specification of Mental Image Retrieval (MIR) as multi-round known-item search guided by a user's internal mental image.
  • GenIR framework: Generative visual feedback pipeline—diffusion-based synthetic image → image-to-image retrieval—making the system's understanding directly inspectable.
  • Dataset: Automated multi-round MIR dataset spanning general objects (COCO), faces (FFHQ), and clothing (Clothing-ADC), with per-round synthetic image, retrieved image, and correctness labels; shown to produce higher-quality queries than ChatIR annotations.
  • Model-agnostic design: Demonstrated plug-and-play compatibility across five state-of-the-art diffusion models with minimal performance variance attributable to generator choice.

Results

  • MSCOCO (50k search space, Hits@10): GenIR (Infinity) achieves ~90% at round 1 and ~98% at round 10; Prediction Feedback reaches 92%, Gemma3-12b Verbal Feedback 92%, ChatIR 73% after 10 rounds.
  • FFHQ: GenIR 70% vs. 52% for next-best baseline.
  • Clothing-ADC (1M+ search space): GenIR 73% vs. 50% for next-best baseline.
  • Flickr30k: GenIR maintains 8–15% advantage over all baselines across all rounds.
  • VLM scale ablation: GenIR with Gemma3-4B outperforms both Prediction Feedback and Verbal Feedback using Gemma3-12B, demonstrating that visual feedback compensates for smaller model size.
  • Generator robustness: Even the weakest generator (HiDream) in GenIR exceeds all non-generative baselines; Infinity and Lumina perform best.
  • Human evaluation: 86% of generated visual feedback rated useful for query refinement by human annotators.

Limitations

  • VLM simulation assumes users hold a clear, fixed target image; real users often begin with partial or fuzzy mental representations.
  • The framework does not model how a user's own mental image naturally clarifies or evolves during the search process.
  • Generated synthetic images can introduce hallucinations or detail misalignments that mislead the refinement rather than helping it.
  • No live human-in-the-loop evaluation beyond the 86% usefulness study; full-scale human-subject experiments are left to future work.

Relevance to Vision-Language Models

GenIR directly exploits the CLIP vision-language embedding space as both retrieval backbone and generator-feedback substrate, making VLM encoding quality a first-order design parameter. The work exposes a fundamental limitation of cross-modal (text→image) embedding matching—text edits cause unpredictable retrieval shifts—and addresses it by projecting queries into the image modality via diffusion, aligning with broader VLM research on grounding textual representations in visual space. For researchers tracking VLMs, this establishes a new interactive retrieval benchmark where VLM query-generation capability (tested with Gemma3-4B/12B) is the core variable, offering a principled testbed for studying how well VLMs translate visual intent into retrieval-effective language. It also connects to compositional and fine-grained VLM understanding, as GenIR's advantages are largest on domains requiring precise attribute discrimination (faces, fashion) that pure text queries handle poorly.