CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception¶

🕒 Published (v1): 2026-05-22 14:07 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; cognitive visual search for high-resolution image perception in MLLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CVSearch is a training-free adaptive framework that resolves the coverage-efficiency dichotomy in high-resolution (HR) image perception for MLLMs via a cognitive "Assess-then-Search" workflow. It dynamically selects between fast visual-expert-assisted search (SAM 3) and a novel semantic-aware scanning fallback, achieving state-of-the-art accuracy on HR benchmarks while maintaining competitive throughput versus scan-based methods.

Problem¶

Current MLLMs process images at fixed, low resolutions (e.g., $336 \times 336$), causing irreversible detail loss for small or occluded objects. Existing remedies face a hard tradeoff: visual-expert-assisted search (SEAL, DyFo) is efficient but creates blind spots when the external expert fails to localize targets; scan-based search (RAP, ZoomEye) guarantees coverage via rigid grid tiling but wastes compute on semantically empty backgrounds and fragments objects that span grid boundaries ("semantic sawtooth" effect).

Method¶

CVSearch implements a three-stage cognitive Assess-then-Search loop:

Global Assessment. The MLLM computes an information-sufficiency score $c_q(I) = M(\text{"Yes"} \mid p_q(Q), I)$ — the model's confidence that the downsampled global view is sufficient to answer query $Q$. If $c_q \geq \tau_q$ (set to 0.9), the model answers directly.
Visual Expert Assisted Search. When $c_q < \tau_q$, the query is parsed into target objects $O = \{o_1, \ldots, o_m\}$, and SAM 3 is queried for bounding boxes $B_e$ and dense visual features $H_e$. If the number of segmented categories matches $|O|$, the crops are passed to the MLLM for answer generation.
Scene-aware Scanning (triggered when $B_e = \emptyset$). Two components:
Semantic Guided Adaptive Patching (SGAP): The pre-extracted features $H_e$ are over-segmented into $N$ superpixels via SLIC on the feature space, then grouped into $k^*$ semantic clusters via Agglomerative Clustering on the region-adjacency graph, optimizing $$k^* = \arg\min_{k \in [k_{\min}, k_{\max}]} \underbrace{L_o(B_k)}_{\text{overlap penalty}} - \underbrace{L_s(H_a, l_k)}_{\text{silhouette score}}.$$ This avoids rigid grid cuts and reuses already-computed expert features.
Dynamic Bottom-Up Search: An adaptive image tree $T$ of depth $D$ is built by recursively applying SGAP. Leaf nodes (maximum detail) are visited first. Each node $n_{d,t}$ is scored by $$c_x = \alpha \cdot c_v + \beta \cdot c_o + \gamma \cdot c_x^*,$$ where $c_v$ is a Visual Complexity prior (average cosine dissimilarity of atomic features within the patch), $c_o$ is the MLLM's object-existence confidence, and $c_x^*$ is the maximum child priority. Nodes with $c_v < \tau_v$ are pruned. The stopping threshold $\tau_{\text{curr}}$ decays adaptively from $\tau_q$ toward minimum $\hat{\tau}_q$ to handle difficult cases without accepting evidence-free regions. On full-tree failure, the top-ranked depth-1 node is fed back to the expert for iterative refinement.

Key Contributions¶

CVSearch framework: first training-free method to unify expert-assisted and scan-based search via a failure-aware cognitive switching mechanism.
SGAP: repurposes failed expert features ($H_e$) for semantically coherent image partitioning, eliminating object fragmentation from rigid grids.
Dynamic Bottom-Up Search: prevents top-down error propagation by starting at leaf nodes; adaptive threshold decay recovers subtle targets without spurious acceptance.
Visual Complexity prior $c_v$: quantifies foreground richness via intra-patch feature dispersion to prune uninformative background branches.

Results¶

V* Bench (avg. $2246 \times 1582$ px): LLaVA-OV-7B 75.4 → 91.6; Qwen2.5-VL-7B 71.2 → 90.1; InternVL2.5-8B 69.1 → 89.0 — rivaling closed-source GPT-4o (66.0) and QWen-VL-max (58.5).
HR-Bench 8K (InternVL2.5-8B): +20.2% overall (57.4 → 77.6); outperforms RAP (74.0) and ZoomEye (73.6) on the same backbone.
HR-Bench 4K (InternVL2.5-8B): +11.0% overall (66.0 → 77.0).
Across all three baselines (LLaVA-OV-7B, Qwen2.5-VL-7B, InternVL2.5-8B), CVSearch consistently outperforms both ZoomEye and RAP on V*, HR-4K, and HR-8K.
Real-world benchmarks (MME-RealWorld-Lite, TreeBench, FineRS-4K): consistent gains, e.g., LLaVA-OV-7B TreeBench 37.3 → 38.8, Qwen2.5-VL-7B 37.0 → 40.7.
Efficiency: $\times 2.9$ throughput improvement over RAP; $\times 1.5$ over ZoomEye (per Figure 1c on Qwen2.5-VL-7B, built-in figure claims).

Limitations¶

Depends on SAM 3 as the visual expert; domains where SAM 3's features are uninformative (e.g., purely textual or abstract image regions) may degrade SGAP quality.
Several hyperparameters ($\tau_q$, $\hat{\tau}_q$, $\tau_v$, $k_{\min}$, $k_{\max}$, $\alpha$, $\beta$, $\gamma$, $D$) are fixed empirically and may require re-tuning for new domains.
Inference overhead remains significantly higher than direct answering; not suitable for latency-sensitive deployment.
Performance on general (non-HR) benchmarks is not reported, leaving open whether the approach regresses on standard tasks.
Table 4 data (Qwen3-VL variants) is truncated in the provided text, preventing full evaluation of generalization to newer model families.

Relevance to Vision-Language Models¶

CVSearch directly targets the high-resolution perception bottleneck — one of the most practically limiting constraints in deployed VLMs — without requiring any additional training or architectural changes, making it a plug-in enhancer for any existing MLLM. The cognitive framing (nonselective → selective attention, failure-aware mode switching) mirrors emerging themes in active and agentic VLM perception. The use of intermediate expert features ($H_e$) as semantic priors for image partitioning is a lightweight form of self-supervised spatial grounding that sidesteps token-count explosion from AnyRes tiling. For researchers tracking VLMs, this establishes a strong training-free upper bound for HR benchmarks and introduces Visual Complexity as a reusable, compute-free signal for spatial attention allocation.