Skip to content

Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VDRP is a prompt learning framework for zero-shot Human-Object Interaction (HOI) detection that addresses two visual complexity challenges: intra-class diversity of verb classes and inter-class visual entanglement between similar verbs. It injects group-wise visual variance and Gaussian perturbation into CLIP text prompt embeddings, then augments those embeddings with LLM-generated, region-specific (human/object/union) concepts retrieved via Sparsemax attention.

Problem

Existing zero-shot HOI detection methods using CLIP prompt learning rely on a single static prompt per verb class. This fails because (1) verb classes exhibit high intra-class visual diversity (diversity score 0.364 ± 0.060 vs. 0.274 ± 0.048 for objects), and (2) semantically distinct verbs (e.g., "eating," "licking," "sitting at") produce visually entangled patterns that a single union-region embedding cannot discriminate.

Method

VDRP is a two-stage HOI detection framework built on frozen DETR + CLIP ViT-B/16:

  1. Visual Diversity-aware Prompt Learning (VDP): For each verb v, union-region CLS features are extracted offline. Verb v's variance σ²_v is pooled with semantically similar verbs (group of 5 by CLIP text embedding cosine similarity) to form a group-wise variance σ̄²_v. An MLP maps σ̄²_v to a modulation vector added (scaled by α=0.02) to the shared learnable context embedding E. The CLIP text encoder produces prompt tv, which is then Gaussian-perturbed: t̃_v = tv + (ε ⊙ σ̃_v)β, encouraging coverage of the verb's appearance distribution.

  2. Region-aware Prompt Augmentation (RAP): LLMs (LLaMA-7B + GPT-4) generate K region concepts (human, object, union) per verb. Concepts are encoded by the CLIP text encoder into concept pools. Given a query region feature x_(·), cosine similarities to pooled concepts are computed and weighted via Sparsemax (to suppress irrelevant concepts). The resulting concept vector c̄v(·) is added to t̃v (scaled by γ=0.2) to yield final region-aware prompts t̂_v(·) for each of the three regions. Final HOI logit = mean of three region logits.

Training uses focal loss for multi-label verb classification.

Key Contributions

  • Group-wise visual variance estimation from CLIP CLS features to modulate verb-specific context embeddings, with Gaussian perturbation scaled by that variance.
  • Region-aware prompt augmentation via LLM-generated concept pools and Sparsemax retrieval, producing separate human/object/union prompts per verb.
  • Unified VDRP framework achieving state-of-the-art on all four HICO-DET zero-shot settings with only 4.50M trainable parameters.

Results

  • NF-UC: 36.45 Unseen mAP, 33.85 HM — surpasses CLIP4HOI by +5.01 Unseen, +4.31 HM.
  • RF-UC: 31.29 Unseen mAP, 32.77 HM — surpasses EZ-HOI by +2.27 Unseen, +1.59 HM.
  • UO: 36.13 Unseen mAP, 34.41 HM — surpasses EZ-HOI by +2.27 HM.
  • UV: 26.69 Unseen mAP, 29.80 HM — surpasses EZ-HOI by +1.59 Unseen.
  • Achieves best results with 4.50M parameters vs. EZ-HOI (6.85M) and CLIP4HOI (56.7M).
  • Ablation confirms both VDP and RAP are individually beneficial and complementary; Sparsemax retrieval outperforms Softmax and Top-3.

Limitations

  • Relies on LLM-generated region concepts (LLaMA-7B + GPT-4) that can introduce noise, particularly for object-region concepts (GPT-4 added to compensate).
  • Group-wise variance requires pre-computing union-region CLS features over the full training set; generalizing variance estimates to truly unseen verbs depends on semantic grouping quality.
  • Evaluated exclusively on HICO-DET; generalization to other HOI benchmarks (e.g., V-COCO, Bongard-HOI) is untested.
  • γ sensitivity: excessive concept weighting (γ=1.0) degrades performance, indicating fragility to noisy retrieved concepts.

Relevance to Vision-Language Models

VDRP directly extends CLIP-based prompt learning—the dominant paradigm for adapting VLMs to downstream vision tasks—to the compositionally challenging domain of HOI detection, where standard single-vector text prompts are demonstrably insufficient. It builds on the CoOp/CoCoOp/MaPLe lineage by introducing distributional (variance-aware) prompt representations, a direction with broad implications for any VLM application involving high intra-class visual variability (e.g., action recognition, fine-grained attribute detection). The retrieval-augmented region prompting connects to RAG-for-vision trends and shows that grounding text prompts in localized, LLM-generated visual concepts—separately for sub-regions—improves compositional generalization in CLIP without large parameter overhead.