HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

HopaDIFF introduces Referring Human Action Segmentation (RHAS), a new task where a natural-language description specifies which person's actions to segment in untrimmed multi-person videos. The authors contribute RHAS133, the first dataset for this task, and a dual-branch diffusion model that fuses holistic video context with person-cropped partial features via a novel HP-xLSTM module and DFT-based frequency conditioning. HopaDIFF substantially outperforms all adapted baselines on RHAS133.

Problem¶

Existing action segmentation methods assume single-person videos with fixed procedural action sequences. They cannot handle untrimmed multi-person videos where the target individual must be identified via a natural-language reference, and they lack the fine-grained temporal control needed to segment spontaneous, unscripted actions across diverse subjects.

Method¶

HopaDIFF is a dual-branch diffusion model operating on VLM-extracted frame features (BLIP-2 or CLIP). The holistic branch encodes the full video with textual reference through the VLM (Fϕ) and an ASFormer encoder (Eh). The partial branch first runs GroundingDINO on each frame to detect the referred person, crops and resizes those regions into a new video stream, then passes it through the same VLM and a separate encoder (Ep).

The two branches exchange information via HP-xLSTM: a modified mLSTM block where the input gates of both branches are computed jointly through bidirectional cross-attention (BCA), allowing structured, selective feature mixing across holistic and partial streams over the full temporal sequence.

Enhanced features ẑh and ẑp are then transformed via Discrete Fourier Transform (DFT) along the temporal axis to produce frequency-domain conditions ẑhf and ẑpf. Each branch's denoising decoder receives both the spatial/temporal embedding and the DFT condition. During inference, action segmentation predictions from both decoders are averaged. Training uses binary cross-entropy and temporal boundary loss equally weighted.

Key Contributions¶

First formulation of Referring Human Action Segmentation (RHAS): multi-person, untrimmed video action segmentation guided by free-form textual person descriptions.
RHAS133 dataset: 133 movies, 542 annotated individuals, 137 fine-grained action classes (extending AVA's 80), 33h of third-person video, with manually cross-validated textual references and multi-label frame-level annotations.
HopaDIFF framework: dual-branch holistic+partial diffusion model with HP-xLSTM for cross-branch temporal reasoning and DFT-based frequency conditioning for fine-grained generative control.

Results¶

All numbers are on RHAS133 test set.

Random partition, BLIP-2 features: - HopaDIFF: ACC=62.58%, EDIT=7.75%, F1@10=87.96%, F1@25=85.50%, F1@50=79.39% - Best baseline (ActDiff): ACC=41.85%, F1@10=70.56%, F1@50=63.29% - HopaDIFF improves F1@50 by ~+16pp over best baseline

Cross-movie evaluation, BLIP-2 features (harder generalization split): - HopaDIFF: ACC=59.63%, EDIT=19.37%, F1@10=90.91%, F1@25=90.33%, F1@50=89.26% - Best baseline F1@50: FACT=57.13%, LTContent=42.55%, ActDiff=21.80% - HopaDIFF improves F1@50 by ~+32pp over best baseline

CLIP features, random partition: - HopaDIFF: ACC=46.62%, F1@50=73.16% vs. best baseline FACT F1@50=49.72%; BLIP-2 features consistently outperform CLIP across all methods

Ablations (cross-movie, BLIP-2): - Removing partial branch (GroundingDINO): F1@50 drops 89.26% → 49.42% - Removing HP-xLSTM: F1@50 drops → 83.50% - Removing DFT conditioning: F1@50 drops → 86.38% - Removing bidirectional cross-attention (BCAF): F1@50 drops → 81.38%

Limitations¶

Two-stage pipeline: relies on pre-extracted VLM features; end-to-end training is not explored.
GroundingDINO detection quality directly gates partial branch quality; failures in person detection propagate to segmentation.
EDIT scores remain low across all methods due to the fine-grained, multi-label nature of RHAS annotations, making boundary precision inherently difficult.
Evaluated only on RHAS133; no cross-dataset transfer experiments beyond cross-movie splits.
Inference cost not fully analyzed; GFLOPS/latency reported only in a table referenced but not included in the provided text.

Relevance to Vision-Language Models¶

VLMs (BLIP-2, CLIP) are not peripheral here — they are the primary feature extraction backbone, and the paper explicitly benchmarks BLIP-2 vs. CLIP, showing that richer cross-modal VLM representations directly translate to substantially better segmentation across all methods. The task itself is a VLM-grounded temporal understanding problem: natural-language person references must be resolved against visual content to drive frame-level prediction over long untrimmed videos. This connects to the growing VLM literature on referring video understanding (referring segmentation, grounding), but extends it to the more demanding regime of per-frame multi-label action recognition in unconstrained multi-person video. The work highlights a current gap: even strong VLM backbones yield poor action segmentation when used naively, motivating architecture innovations (like HP-xLSTM) that explicitly exploit the holistic-partial duality.