MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MSTAR is a box-free scene text retrieval framework that eliminates bounding-box annotation requirements by building on pretrained VLMs (BLIP-2 + SigLIP) augmented with Progressive Vision Embedding (PVE) to shift attention toward fine-grained, low-salience text regions. It unifies four query types (word, phrase, combined, semantic) through style-aware instructions and a Multi-Instance Matching module. It also introduces MQTR, the first benchmark covering all four query types across 16k images.

Problem

Existing scene text retrieval methods require expensive bounding-box annotations (word-level, line-level, or object-level) for training. They also use query-type-specific retrieval strategies that cannot generalize across diverse query forms. Pretrained VLMs used off-the-shelf tend to ignore low-salience text instances (e.g., small or peripheral text), causing high miss rates in fine-grained retrieval.

Method

MSTAR is built on a SigLIP ViT-Base-512 vision encoder and a BLIP-2 multi-modal encoder (Q-Former). Three core modules are stacked on top:

1. Progressive Vision Embedding (PVE): At each iteration \(t\), the cross-attention map \(C_{t-1}\) from the Q-Former is binarized via a thresholding + watershed + connected-components pipeline (\(\sigma\)) and inverted into a binary mask \(M_{t-1}\). A masked multi-head self-attention (Salient Attention Shift, SAS) then suppresses the already-attended salient regions in image features \(f_{t-1}\), forcing subsequent Q-Former queries onto neglected regions. Vision embeddings from \(T+1\) iterations are concatenated: \(E_V \in \mathbb{R}^{(T+1)Q \times d}\).

2. Style-aware Instructions: Each query type (word/phrase/combined/semantic) is prepended with a short type-specific instruction token before encoding with the Q-Former, embedding queries into distinct representation subspaces without separate encoders.

3. Multi-Instance Matching (MIM): Two parallel branches handle alignment. For single-word queries, Hungarian matching assigns one-to-one correspondences between word embeddings \(E_w\) and vision embedding slots. For multi-word queries, a lightweight cross-attention layer aggregates vision features under text constraint. Training uses dual contrastive loss \(L_c = \alpha L_{t2v} + L_{v2t}\) (with \(\alpha=1.5\) to balance query/image counts) plus image-text matching loss \(L_m\). Inference uses cosine similarity ranking followed by re-ranking of top 2% with the full Q-Former.

Key Contributions

• First box-free scene text retrieval model, eliminating word/line/object bounding-box annotation costs.
• Progressive Vision Embedding with Salient Attention Shift to iteratively redirect VLM attention from salient to fine-grained text regions without positional supervision.
• Unified multi-query support (word, phrase, combined, semantic) via style-aware instructions within a single model.
• MQTR benchmark: 16k images, four query types, 625 queries, including hard negatives with visual/textual confounders — the first benchmark covering combined and semantic query types for scene text retrieval.

Results

• MQTR overall: MSTAR 66.78% MAP average vs. best prior box-based TG-Bridge 54.09% (+12.69%) and fine-tuned BLIP-2 58.11% (+8.67%); surpasses previous models by 8.53% average MAP.
• Phrase retrieval (PSTR dataset): MSTAR 95.71% MAP vs. FDP 92.28% (+3.43%), TDSL 89.40%.
• Word retrieval (6 public datasets average): MSTAR 82.56% MAP vs. TDSL 74.16% (+8.40%); +re-rank: 84.18%.
• Total-Text word retrieval: MSTAR 85.55% vs. FDP 79.18% (+6.37%).
• CTW: MSTAR 90.87% vs. TDSL 59.34% (+31.53%).
• Inference speed: 14.2 FPS vs. TG-Bridge 6.7 FPS (>2× faster); TDSL 12.0 FPS.
• Ablation (Tab. 6): Adding PVE to baseline+Ins+MIM improves CTR by 4.36%, IC15 by 1.98%.
• Limitation case: CTR (extreme small text) — MSTAR underperforms TDSL due to lack of positional supervision.

Limitations

• Still lags behind box-based methods on extremely small and densely packed text instances, where precise positional supervision provides an irreplaceable advantage.
• Re-ranking the top 2% introduces a secondary inference pass, reducing throughput (14.2 → 6.9 FPS with re-ranking).
• PVE iterative steps beyond \(T=1\) improve accuracy but reduce speed; the paper fixes \(T=1\) as a trade-off.
• MQTR semantic subset is small (1,000 images, 25 queries) and manually collected, potentially limiting generalization assessment for semantic retrieval.

Relevance to Vision-Language Models

MSTAR directly investigates a known failure mode of contrastively pretrained VLMs (CLIP, BLIP-2, SigLIP): their cross-attention concentrates on salient high-level objects and ignores fine-grained text instances embedded in scenes. The PVE mechanism provides a task-specific, supervision-free fix — recycling the model's own attention maps to iteratively redirect focus — which is a broadly applicable insight for any VLM application requiring localized, low-salience feature extraction. The work also demonstrates that prompt-style style-aware instructions suffice to unify semantically heterogeneous query types in a single VLM encoder, avoiding task-head proliferation. For researchers tracking VLMs, this paper exemplifies how to adapt frozen or lightly fine-tuned VLM backbones to structured recognition tasks without bounding-box supervision, connecting to the broader theme of weakly supervised grounding in document and scene understanding.