Skip to content

HKVLM: Faithful Reasoning Grounding by Binding Language Queries to a Frozen Detector

🕒 Published (v1): 2026-06-27 11:10 UTC · Source: Arxiv · link

Why this paper was selected

Practical binding of LM reasoning to frozen detector; addresses VLM faithfulness directly

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs "see but mis-speak" because localization passes through the autoregressive head, entangling geometry with language generation. HKVLM breaks this by binding referential query embeddings from a frozen LLM to region proposals from a frozen detector via a lightweight contrastive alignment hook, plus a perception-grounded faithfulness veto that suppresses ungrounded assertions. With only the hook trained (frozen Grounding DINO + Qwen2.5-VL), grounding accuracy jumps 50–90× over untrained cross-space matching and the veto cuts hallucination rate from ~0.99 to 0.23–0.43 on POPE.

Problem

Coordinate-as-text VLMs route bounding-box generation through the autoregressive decoder (as coordinate tokens or quantized symbols), coupling localization to language generation. A single bad query token can override correct perception. Two-stage reasoning pipelines (e.g., DetGPT, LISA) separate localization but compress the model's intent into a single class string, discarding instance-specific and relational cues. The result is a binding failure: the model attends to the right pixels yet returns the wrong box or label. No existing design has an explicit objective that ties language semantics to region geometry or that penalizes ungrounded assertions.

Method

HKVLM removes localization entirely from the language path with four components:

  1. Frozen perception module — a language-aligned open-vocabulary detector (Grounding DINO) emits \(M\) class-agnostic proposals \(R = \{(b_j, e_j)\}_{j=1}^M\) with boxes \(b_j\) and vision-language-fused region embeddings \(e_j \in \mathbb{R}^d\).
  2. Referential query embeddings — a frozen LLM (Qwen2.5-VL-7B) reads image \(I\) and request \(q\), producing \(K\) continuous query embeddings \(\{u_k\}_{k=1}^K\) from dedicated query slots, preserving more intent than a class string.
  3. Alignment hook — two light projections map into a shared space: \(\tilde{e}_j = f_\theta(e_j)\), \(\tilde{u}_k = g_\phi(u_k)\). Binding uses contrastive retrieval (pulling each query toward its matched regions, pushing from the rest) plus bipartite Hungarian assignment that reuses DETR's set-prediction machinery to resolve which proposal each query claims.
  4. Perception-grounded faithfulness veto — a verification head \(v(\cdot)\) predicts whether the best-matched pair is genuinely supported. At inference, a category is emitted only when its bound region passes the verifier and affinity exceeds a threshold; otherwise the model abstains.

Only \(\theta, \phi\), a box-refinement residual, and the verifier are trained — perception and the LLM remain frozen, targeting small-data cold-start settings.

Key Contributions

  • A diagnostic say-vs-see decomposition separating localization error (SeeErr — the right region was never proposed) from binding error (SayErr — the right region was available but mis-bound or misnamed).
  • HKVLM framework in which a frozen detector owns localization and an LLM's referential queries are bound to region embeddings by shared-space alignment rather than autoregressive coordinate/index tokens.
  • A perception-grounded faithfulness objective and veto that suppresses categories no region supports, directly targeting object hallucination.
  • A frozen-stack training recipe for small-data, cold-start adaptation where low-rank tuning of a monolithic VLM typically stalls.

Results

  • Referring grounding (RefCOCO/+/g, IoU 0.5, \(M=300\)): HKVLM achieves 0.254–0.270 grounding accuracy vs. 0.003–0.005 for naive cosine at the same frozen perception — a 50–90× gain. Binding accuracy reaches 0.123–0.148 (vs. ≤0.005).
  • POPE object hallucination: Without the veto, accuracy is near-chance (~0.50) with hallucination rate ~0.98–0.99. The faithfulness veto lifts POPE accuracy to 0.762/0.718/0.661 (random/popular/adversarial) and reduces hallucination to 0.23/0.32/0.43.
  • Grounding vs. faithfulness decoupling: removing the veto changes grounding accuracy by ≤0.01, confirming the veto adds faithfulness without substantially altering localization.
  • Data efficiency: training only the hook reaches 0.139 grounding accuracy at 200 expressions and 0.266 at 24k — 200 expressions already outperform naive cosine by 46×.
  • Proposal count effect: increasing \(M\) from 50 to 300 improves grounding by 19–24% without retraining, confirming SeeErr (not SayErr) is the residual bottleneck.
  • Controlled binding study (perception assumed perfect): HKVLM hook achieves 96.0% binding accuracy vs. 83.5% for class-string interface and 7.1% for naive cosine.

Limitations

  • Absolute grounding accuracy (~0.26) is capped by frozen-proposal recall under a generic detection prompt — HKVLM cannot bind to an object the detector never proposed (dominant SeeErr).
  • Larger proposal pools (\(M=300\)) increase false-positive rate of the veto (hallucination rises from ≤0.14 at \(M=50\) to 0.23–0.43), suggesting the faithfulness threshold should be jointly optimized with \(M\).
  • Evidence is confined to two ablation references sharing the frozen perception; head-to-head comparison against external systems (DetGPT, ChatRex, coordinate-as-text VLMs) and against low-rank VLM tuning at equal data budget are reserved for future work.
  • The controlled simulation (Section 7) assumes perfect perception (SeeErr = 0), so its strong binding numbers do not transfer directly to real-image settings.

Relevance to Vision-Language Models

HKVLM directly tackles the VLM grounding and hallucination problems that plague open-source models adapted for detection tasks. Its explicit decoupling of localization from language generation — and the formal SeeErr/SayErr decomposition — provides both a diagnostic tool and a practical architecture for practitioners who need faithful, reasoning-capable grounding without monolithic VLM fine-tuning. The finding that a lightweight alignment hook can bind frozen components 50–90× better than naive cross-space matching, and that an explicit faithfulness veto (not grounding alone) is what suppresses hallucination, are concrete results that should inform the design of next-generation grounded VLMs.