Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild

🕒 Published (v1): 2026-07-15 14:30 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentHOI is a training-free agentic framework for human-object interaction (HOI) detection that orchestrates a multimodal LLM (MLLM) perception module and a spatial grounding model without any task-specific fine-tuning. Two plug-and-play mechanisms — Context-aware Multi-round Reasoning and Multifaceted Interaction Localization — address incomplete interaction discovery and ambiguous spatial grounding in complex scenes. The framework outperforms supervised and weakly-supervised zero-shot baselines on HICO-DET under both standard and distribution-shifted evaluations.

Problem

Existing HOI detectors (supervised, open-vocabulary, weakly-supervised) couple interaction understanding to fixed label spaces and dataset-specific visual distributions, limiting generalization to open-world scenes with novel interaction compositions or shifted image styles. Recent MLLM-based approaches use these models only as feature extractors or semantic aligners, underexploring their compositional reasoning potential. Two concrete failure modes remain unaddressed: (1) single-pass MLLM queries miss interactions in multi-action scenes (recall drops 14.92%, F1 drops 2.83% in multi-HOI vs. single-HOI scenes); (2) generic grounding phrases fail to disambiguate visually similar instances.

Method

AgentHOI decomposes HOI detection into two coordinated modules:

AgentHOI-Base: An MLLM \(\mathcal{M}_L\) receives image \(I \in \mathbb{R}^{H \times W \times 3}\) with object/action prior sets \(\mathcal{C}_o, \mathcal{C}_a\) and produces action-object pairs \(\{(c_a, c_o)\}\). A grounding model \(\mathcal{M}_g\) then localizes each pair using template phrases to retrieve bounding boxes \((b_h, b_o)\).

Context-aware Multi-round Reasoning extends this with three sequential MLLM queries: 1. Initial HOI identification: \(T_1 = \mathcal{M}_L(I, P_1)\) — standard salient interaction extraction. 2. HOI re-mining: \(T_2 = \mathcal{M}_L(I, P_2, T_1)\) — \(T_1\) used as contextual anchors to prompt the model to recover missed interactions. 3. Action reassignment: \(T_3 = \mathcal{M}_L(I, P_3, \{T_1, T_2\})\) — a fine-grained object-specific action prior \(d_o\) guides reassessment of action categories for all identified pairs.

Multifaceted Interaction Localization (MIL) replaces generic grounding phrases with MLLM-generated instance-specific descriptions \(D_h\) (e.g., "left person in black shirt carrying a bottle") and \(D_o\), forming rich referential queries. Grounding confidence is fused with MLLM classification confidence as: $\(s_{a,o}^{\text{final}} = s_{a,o} \cdot (s_h \cdot s_o)^\lambda\)$ where \(s_{a,o} = \frac{p_a + p_o}{2}\) and \(\lambda\) balances localization reliability.

Key Contributions

• First training-free agentic HOI detection framework using a large-scale MLLM, requiring zero HOI-specific annotations or parameter updates.
• Context-aware Multi-round Reasoning: a three-stage iterative querying harness (identify → re-mine → reassign) that improves interaction completeness in compositional scenes.
• Multifaceted Interaction Localization: descriptive prompting strategy that generates instance-specific textual cues to disambiguate grounding among visually similar entities.
• New evaluation protocol applying style transfer and image quality degradation to HICO-DET test images, enabling distribution-shift robustness assessment.

Results

All numbers are mAP on HICO-DET zero-shot splits (UV = unseen verb, UO = unseen object, RF-UC = rare-first unseen composition, NF-UC = non-rare-first unseen composition):

• Original test set: AgentHOI achieves UV 29.85, UO 33.70, RF-UC 38.64, NF-UC 28.04 — best on UV and RF-UC vs. all supervised zero-shot methods (e.g., HOLa: UV 27.91, RF-UC 30.61; CMMP: UV 26.23, RF-UC 29.45).
• Style-transferred: AgentHOI UV 26.45, UO 28.34, RF-UC 32.55, NF-UC 23.05 — supervised methods drop substantially (e.g., HOLa UV falls to 18.83, CMMP UV to 17.32).
• Degraded images: AgentHOI UV 27.61, UO 28.99, RF-UC 34.79, NF-UC 24.56 — again more robust than supervised counterparts.
• Weakly supervised comparison (default HICO-DET): AgentHOI achieves 29.61 mAP overall and 40.33 mAP on the Rare split, outperforming all weakly-supervised methods by a large margin.
• NF-UC exception: Supervised zero-shot methods marginally outperform AgentHOI here; analysis attributes this to no_interaction annotation incompleteness (17 of AgentHOI's bottom-20 NF-UC categories involve no_interaction).

Limitations

• Performance on NF-UC lags behind supervised zero-shot methods because AgentHOI's open-ended reasoning may predict valid interactions that are not exhaustively annotated in the dataset, artificially depressing measured mAP.
• The three-round MLLM querying plus per-instance descriptive prompting introduces substantial inference-time compute overhead relative to single-forward-pass detectors.
• Relies on a fixed object/action prior vocabulary (\(\mathcal{C}_o, \mathcal{C}_a\)) supplied as prompt context; truly unconstrained open-world discovery without any prior sets is not demonstrated.
• Evaluated solely on HICO-DET; generalization to other HOI benchmarks (V-COCO, HOI-A) is not shown.
• The balancing coefficient \(\lambda\) in the confidence fusion formula requires tuning without a validation set in training-free deployments.

Relevance to Harnesses / Meta-Harnesses

AgentHOI is a concrete instantiation of a multi-module agentic harness: it hard-codes a fixed orchestration loop over heterogeneous specialist models (MLLM + grounding model), with structured control flow (three sequential query rounds with anchor-passing between rounds) that mirrors harness patterns like chain-of-thought scaffolding and iterative re-querying. The Context-aware Multi-round Reasoning is essentially a prompt-level meta-harness that routes intermediate outputs back as context for subsequent LLM calls — a reusable design pattern applicable beyond HOI detection. The clean separation between a reasoning orchestrator (MLLM) and a specialist executor (grounding model), with no shared fine-tuning, demonstrates how harnesses can compose foundation models without task-specific supervision, a key claim in the meta-harness literature.