Skip to content

Guided Query Refinement: Multimodal Hybrid Retrieval with Test-Time Optimization

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Test-time query refinement bridges modality gap in multimodal retrieval with VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Guided Query Refinement (GQR) is a test-time optimization method that refines a vision-centric retriever's query embedding using gradient descent guided by similarity scores from a lightweight complementary text retriever. It closes the performance gap with much heavier models while being up to 14× faster and requiring 54× less memory, pushing the Pareto frontier for multimodal document retrieval.

Problem

State-of-the-art visual document retrievers (ColPali-based, late-interaction) achieve strong performance by operating directly on image patches, but at enormous representational cost—e.g., LLAMA-NEMO-RETRIEVER-COLEMBED-3B stores ~10 MB per document page. Additionally, purely vision-centric models remain constrained by the modality gap in modern VLMs. Standard hybrid retrieval (rank-level or score-level fusion) only performs coarse aggregation and cannot exploit the rich intra-space query–document interactions of each retriever.

Method

GQR operates as a two-stage test-time procedure with a primary (vision-centric, ColPali-based) and a complementary (lightweight text-based) retriever:

  1. Candidate pool construction: both retrievers independently retrieve top-\(K\) documents; their union forms \(\mathcal{C}(q)\).
  2. Query refinement: starting from the primary retriever's query embedding \(z^{(0)} = e^q_1\), GQR iterates for \(T\) steps. At each step \(t\) it computes a consensus distribution over \(\mathcal{C}(q)\): $\(p^{(t)}_{\text{avg}}(d) = \frac{1}{2}\left[p_1(d \mid z^{(t)}) + p_2(d \mid e^q_2)\right]\)$ and minimizes the KL divergence \(\mathcal{L}^{(t)} = \text{KL}\!\left(p^{(t)}_{\text{avg}} \,\|\, p_1(\cdot \mid z^{(t)})\right)\) via Adam with step size \(\alpha\), updating \(z^{(t+1)} = z^{(t)} - \alpha \nabla_{z} \mathcal{L}^{(t)}\). The final ranking uses \(s_1(z^{(T)}, d)\).

The complementary retriever's distribution \(p_2\) is fixed throughout; only the primary query embedding moves. This constrains the update to remain within the primary model's representational geometry, naturally downweighting a weaker or misaligned secondary signal.

Key Contributions

  • GQR: a novel architecture-agnostic, test-time KL-minimization framework for hybrid retrieval that operates at the representation level without requiring cross-space alignment or supervision.
  • Demonstration that weaker text retrievers (e.g., Qwen3, 16+ NDCG points below ColPali-based primary) still provide useful complementary signal via GQR.
  • Consistent Pareto improvements over both single-retriever and all standard hybrid baselines (RRF, score aggregation) across ViDoRe 1, 2, and 3.
  • Zero-shot generalization: hyperparameters tuned on ViDoRe 2 transfer directly to ViDoRe 3 with consistent gains across all 6 model pairs and all 8 domain subsets.

Results

  • ViDoRe 2 (NDCG@5, tuned setting): GQR achieves average +3.9% gain over primary retriever, outperforming all hybrid baselines (next best: Score Aggregation Min-Max Tuned at +3.4%); standard deviation across 9 pairs is 1.9, lower than most baselines.
  • Efficiency: Colnomic-7B + GQR (Linq) reaches NDCG@5 = 62.7 at 181 ms vs. Llama-Nemo at 62.9 NDCG@5 and 2,591 ms — ≈14× faster, ≈54× less memory.
  • Surpassing SOTA: Colnomic + GQR (Jina) attains NDCG@5 = 63.0 at 350 ms (≈7× faster than Llama-Nemo, ≈24× less memory) while outperforming it outright.
  • ViDoRe 3 (zero-shot): Colnomic-7B + GQR (Jina) improves average NDCG@5 from 55.7 → 57.4 (+1.7); Jina vision + GQR (Linq) improves 53.4 → 55.4 (+2.0); per-subset gains reach +4.4.
  • Hybrid baselines: Ranking-level methods (RRF, Average Ranking) hurt performance on average (−2.8% to −3.0%); GQR is the only method consistently positive across all subsets.

Limitations

  • GQR introduces \(T\) optimization steps at query time, adding latency (192–241 ms for \(T\)=25–50); latency scales with \(T\) and learning rate choices.
  • Hyperparameters \(\alpha\) and \(T\) require tuning on a development set; although zero-shot transfer works, optimal configuration varies per model pair.
  • Evaluation is limited to visual document retrieval (PDF-centric); generalization to other multimodal retrieval tasks (image-text matching, video retrieval) is untested.
  • Llama-Nemo excluded from ViDoRe 3 evaluation due to prohibitive computational cost, limiting the completeness of the largest-model comparison in the zero-shot setting.
  • The secondary retriever's signal can degrade performance in rare cases (e.g., Qwen3 as complementary on specific subsets), indicating the approach is not universally safe without tuning.

Relevance to Vision-Language Models

GQR directly addresses a core weakness of modern VLMs used for retrieval: the modality gap that arises when a VLM-based encoder must match textual queries against visually encoded documents. By injecting complementary uni-modal text signal at test time — without retraining — GQR offers a practical, architecture-agnostic fix applicable to any ColPali-style late-interaction VLM encoder. The work also quantifies the efficiency–accuracy tradeoff for large-scale multimodal encoders, which is increasingly relevant as VLM-based retrievers scale to billions of parameters. For researchers tracking VLMs, this establishes that lightweight text retrievers remain useful complements to powerful vision-language encoders, and that test-time optimization over embedding spaces is a tractable path to bridging the modality gap without architectural surgery.