LINK: Learning Instance-level Knowledge from Vision-Language Models for Human-Object Interaction Detection

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Instance-level VLM knowledge for HOI detection; balances generalization and specialization

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LINK proposes a two-stage HOI detection framework that bridges image-level VLM representations to instance-level human-object interaction reasoning via a Human-Object Geometrical Encoder and a VLM Linking Decoder. A progressive teacher-student learning strategy alleviates sparse HOI supervision by distilling dense guidance from a pre-trained teacher across all candidate human-object pairs, not only GT-matched ones. LINK establishes new state-of-the-art on HICO-DET, V-COCO, and SWiG-HOI across fully-supervised, zero-shot, and open-vocabulary settings.

Problem

Two compounding challenges limit VLM-based HOI detection. (1) Supervision sparsity: HOI datasets annotate only a small subset of valid human-object pairs in a scene, leaving unlabeled positives and informative negatives unused. (2) Specialization–generalization trade-off: architectures optimized for fully-supervised benchmarks degrade under zero-shot evaluation, while zero-shot methods (lightweight CLIP adapters) underperform in fully-supervised settings—no prior architecture excels at both simultaneously.

Method

LINK is a two-stage detector whose key modules decouple from any specific object detector backbone.

Human-Object Geometrical Encoder. Unary queries \(Q_h, Q_o \in \mathbb{R}^C\) are extracted via ROI Align on VLM feature maps (bypassing detector-specific object queries). Each box \(B=(x_1,y_1,x_2,y_2)\) is normalized and encoded as \(\text{PE}(B) = \text{PE}(C) \oplus \text{PE}(S)\) using 2D sinusoidal encodings on center and size, then added to unary queries and refined by self-attention. Pairwise queries are formed by \(Q_{h\text{-}o} = \text{Linear}(\mathcal{C}[Q_i, Q_j])\) for all \((i,j) \in H \times (O \cup H)\). A pairwise spatial relation vector \(R_{i,j}\) (directional vector, relative size, IoU—following UPT) is fused with \(Q_{h\text{-}o}\) via a Multi-Modal Fusion module (two FC projections + LayerNorm + MLP).

VLM Linking Decoder. Dual-branch cross-attention over the VLM feature map: a spatial branch that compresses the feature map to a latent bottleneck with box-encoding-guided attention (\(\text{CA}_{be}\)), and a semantic branch that expands queries into the VLM's native high-dimensional space for rich semantic aggregation. Outputs are concatenated and fused by an MLP. The final classification head is an FFN initialized with CLIP text embeddings for HOI triplet logits.

Progressive Learning Strategy. Stage 1: train the architecture on GT annotations to produce a Teacher Model. Stage 2: a Student Model (same architecture, trained from scratch) receives (i) sparse GT supervision via matched queries \(M\) (IoU ≥ 0.5 with GT boxes) and (ii) dense teacher guidance over all human-object pairs at three levels using KL-divergence distillation (\(\tau=2\)): $\(\mathcal{L}^* = \mathbb{E}_{I}\!\left[\mathcal{L}_M(\Phi_\theta, \text{GT}) + \sum_{g \in \{\text{feat,query,logit}\}} \text{KD}_g(\Phi_\theta, \Phi_t)\right]\)$ Feature-map alignment uses bilinear/trilinear interpolation for spatial and channel matching; query-level distillation averages token-wise KL across all encoder and decoder layers; logit-level distillation operates on final HOI scores combined with detection confidence.

Key Contributions

• Unified HOI detector (HOGE + VLM Linking Decoder) that is plug-and-play with arbitrary object detectors by using ROI Align on VLM features rather than detector-specific object queries.
• Progressive teacher-student strategy providing dense, three-level distillation (feature map, query, logits) over all candidate human-object pairs, not just GT-matched instances.
• First systematic evaluation of HOI detection across six foundation models (CLIP, BLIP, DINOv2, DINO@448, SigLIP2, Florence2), demonstrating consistent gains especially on long-tail interactions (≤10 training samples).
• State-of-the-art across fully-supervised, zero-shot (RF-UC, NF-UC, UO, UV), and open-vocabulary HOI settings on three benchmarks.

Results

• HICO-DET (R50 + ViT-L CLIP): 42.92 / 45.03 mAP (full / rare), surpassing prior SOTA by +3.87 / +6.37 mAP (9.9% / 16.5% relative gain).
• HICO-DET (Swin-L): 49.06 / 53.63 mAP (full / rare).
• Zero-shot RF-UC (HICO-DET): unseen 32.25 vs. prior best (HOLa) 30.61 (+1.64); LINK† (ViT-L) unseen 38.51.
• Zero-shot NF-UC: HM 35.14 (LINK), 40.33 (LINK†).
• Zero-shot UO: HM 37.92 (LINK), 38.24 (LINK†)—new SOTA.
• Zero-shot UV: HM 29.29 (LINK), 31.92 (LINK†); LINK† unseen 38.36 vs. HOLa 27.91.
• Consistently improves all six foundation model baselines, with largest gains on long-tail HOIs.

Limitations

• Two-stage paradigm inherits dependence on a pre-trained object detector; localization errors in the detector propagate to HOI classification.
• The teacher model (frozen CLIP ViT-L/14@336px) transfers its own biases and failure modes to the student.
• Progressive training requires two full training passes, adding wall-clock and memory overhead compared to single-stage methods.
• Dense supervision over all \(|H| \times |O \cup H|\) pairs scales quadratically with the number of detected instances, which may be costly in crowded scenes.
• The text does not report results on open-vocabulary HOI benchmarks with non-COCO object vocabularies or very large-scale real-world deployment.

Relevance to Vision-Language Models

LINK directly tackles the central challenge of lifting image-level VLM representations (CLIP's global contrastive embeddings) to instance-level, spatially-grounded reasoning—a fundamental bottleneck for applying VLMs to structured prediction tasks beyond retrieval. The dual-branch VLM Linking Decoder and the CLIP-initialized logit head provide a transferable pattern for injecting VLM semantics into detection heads without fine-tuning the backbone. The progressive distillation strategy demonstrates that VLM knowledge can be propagated to dense, instance-level supervision, addressing the critical data sparsity problem in annotation-expensive compositional tasks. The cross-backbone evaluation spanning CLIP, BLIP, DINOv2, SigLIP2, and Florence2 offers a rare comparative study of how different VLM pretraining objectives translate to fine-grained spatial tasks.