Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; MLLM as detector-agnostic interaction recognizer for zero-shot HOI

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes DA-HOI, a decoupled framework for zero-shot Human-Object Interaction (HOI) detection that fully separates object detection from interaction recognition (IR) and replaces CLIP-based classifiers with an MLLM (Qwen 2.5-VL 3B) formulated as a VQA task. A deterministic generation method enables training-free IR, while a spatial-aware pooling module and one-pass deterministic matching improve fine-tuned performance and efficiency. The resulting system is plug-and-play with any object detector without retraining.

Problem¶

Existing zero-shot HOI detection methods tightly couple interaction recognition with a specific object detector and rely on CLIP's coarse-grained vision-language features. This coupling prevents independent improvement of the detection and IR components, forces retraining when the detector changes, and limits generalization to unseen interactions because CLIP's feature alignment operates only over training-observed categories.

Method¶

The framework decouples object detection from IR into two independent stages:

Training-free IR via deterministic generation. For each human-object pair \((B_h^i, B_o^i, C_o^i)\), IR is cast as VQA: an MLLM receives image features plus ROIAlign-pooled interaction features \(f_{\text{inter}}^i = \text{Concat}(f_h^i, f_o^i)\) and a candidate interaction list \(\Theta(C_o^i)\). Instead of open-ended generation, confidence scores are computed as the conditional likelihood \(S_v^i[k] = p(T_k \mid I, Q^i) = \prod_{j=1}^N p(t[j] \mid T_k[<j], I, Q^i)\), making the output deterministic and mAP-compatible.
Spatial-Aware Pooling (SAP). Replaces bare ROIAlign features with a cross-attention layer that aggregates image context beyond the bounding box, plus an MLP-projected pairwise spatial embedding encoding box areas, aspect ratios, IoU, and human-to-object direction. An interactiveness classifier \(S_{\text{interactiveness}} = \sigma(\text{Linear}(f_{\text{inter}}))\) prunes non-interactive pairs before LLM inference.
One-Pass Deterministic Matching. A special token <|hoi|> is appended after each candidate in the prompt; cosine similarity between the LLM's output feature for each <|hoi|> token and the interaction feature \(\hat{f}_{\text{inter}}^i\) replaces per-candidate forward passes: \(S_v^i[k] = \text{cosine}(\hat{f}_{\text{hoi}}^i[k], \hat{f}_{\text{inter}}^i)\), reducing \(M\) forward passes to one.

Training uses a two-stage procedure: SAP is trained first with binary focal loss on interactiveness; then the LLM is fine-tuned with LoRA on interaction matching while the visual encoder stays frozen.

Key Contributions¶

Full decoupling of object detection and interaction recognition, enabling plug-and-play compatibility with any detector without retraining.
Deterministic generation formulation that converts MLLM open-ended generation into a likelihood-based confidence scoring mechanism, yielding a strong training-free zero-shot IR baseline.
Spatial-aware pooling (SAP) that combines cross-attention context aggregation with explicit pairwise spatial encoding to handle imperfect detections.
One-pass deterministic matching that collapses \(M\) per-candidate LLM forward passes into a single pass via special token feature matching.

Results¶

All numbers are mAP on HICO-DET unless noted.

RF-UC (Full): Ours 43.56 / Ours† (Grounding-DINO) 44.81 / Ours‡ (YOLO-World) 44.00 vs. BC-HOI 40.99, LAIN 34.41, EZ-HOI 36.73.
NF-UC (Full): Ours 40.33 / Ours† 41.51 / Ours‡ 42.01 vs. BC-HOI 36.40, LAIN 33.23.
UO (Full): Ours 43.60 / Ours† 45.28 / Ours‡ 44.82 vs. BC-HOI 34.18, EZ-HOI 36.38.
UV (Full): Ours 42.88 / Ours† 44.43 / Ours‡ 43.88 vs. BC-HOI 39.89, EZ-HOI 36.84.
Average Full across all four zero-shot settings: Ours 42.59 / Ours† 44.00 / Ours‡ 43.68.
Training-free setting (Full, all splits): Ours⋆ 31.50 vs. ADA-CM⋆ 25.19 — a +6.31 mAP improvement.
Cross-dataset (HICO-DET → V-COCO): outperforms CMMP by 12.26%.

Limitations¶

Evaluation is restricted to HICO-DET and V-COCO; generalization to more diverse HOI benchmarks is not demonstrated.
The interactiveness threshold \(\lambda = 0.15\) and the IoU-based pair assignment during training may still produce noisy labels from imperfect ground-truth box matching.
The candidate interaction list \(\Theta(C_o^i)\) must be predefined per object category; the method does not handle truly open-vocabulary interactions where no candidate list exists.
Computational cost scales with the number of remaining human-object pairs after interactiveness filtering; very crowded scenes may still be slow despite the one-pass matching.
The MLLM backbone (Qwen 2.5-VL 3B) introduces a large parameter count compared to CLIP-based alternatives, and the paper does not report latency or FLOPs.

Relevance to Vision-Language Models¶

This work directly addresses a key failure mode of CLIP-based VLMs — coarse-grained feature alignment that cannot distinguish visually similar interactions — by substituting a generative MLLM used in a discriminative mode. The deterministic generation trick (conditional likelihood scoring) is a broadly applicable technique for adapting any instruction-following MLLM to structured classification tasks without fine-tuning, relevant to practitioners using VLMs for fine-grained recognition. The decoupled architecture demonstrates how MLLMs can serve as modular, detector-agnostic reasoning components within structured perception pipelines, a design pattern increasingly important as open-vocabulary detectors mature. The cross-dataset generalization results provide evidence that MLLM-based IR generalizes substantially better than CLIP-based feature matching.