DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding¶

🕒 Published (v1): 2026-07-01 09:18 UTC · Source: Arxiv · Venue: ECCV · link

Why this paper was selected

ECCV; difficulty-adaptive routing for zero-shot video grounding, strong venue, novel routing idea

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DART is a zero-shot video temporal grounding framework that uses a query-conditioned Determinantal Point Process (DPP) to simultaneously select diverse, query-relevant keyframes and derive a spectral entropy signal that routes queries to either direct prediction (Fast path) or structured chain-of-thought reasoning (Slow path). By activating LVLM reasoning only when query complexity warrants it, DART closes the "reasoning gap" between simple and complex queries that defeats pure feature-matching methods. On Charades-STA and ActivityNet Captions, it achieves state-of-the-art zero-shot performance while using over 7× fewer frames than the strongest baseline.

Problem¶

Existing zero-shot VTG methods compute per-frame feature similarity between query and video frames independently, making them unable to model cross-frame temporal ordering, causal relations, or conditional structure. This causes a systematic 11.6 mIoU degradation on complex queries versus simple ones (42.2% vs. 30.6% for the best prior method on ActivityNet Captions). Naively applying Chain-of-Thought prompting is also problematic: query-agnostic frame sampling either misses event stages or floods the model with irrelevant distractors, and reasoning every query wastes latency while compounding errors on simple inputs.

Method¶

DART proceeds in three tightly coupled stages using LLaVA-1.6-7B as the LVLM backbone.

Query-Conditioned DPP Kernel. Given frame features $F = [f_1, \ldots, f_M]$ and text feature $q$, a base kernel $L_{ij} = k_v(f_i, f_j) \cdot k_t(t_i, t_j)$ combines visual and temporal Gaussian kernels. Per-frame relevance weights $w_i = \sigma(f_i^\top q / \tau)$ produce the query-conditioned kernel $\tilde{L}_{ij} = \sqrt{w_i}\,L_{ij}\,\sqrt{w_j}$. Greedy selection maximizes the marginal log-determinant gain, decomposed as: $$\Delta(e \mid S) = \underbrace{\log\det(\mathbf{L}_{S\cup\{e\}}) - \log\det(\mathbf{L}_S)}_{\text{diversity}} + \underbrace{\log w_e}_{\text{query relevance}}$$ Selection halts adaptively when the relative gain falls below threshold $\Delta_k/\Delta_1 < \varepsilon$, yielding a query-specific keyframe budget $K$.

Spectral Entropy Routing. The eigenvalues $\{\lambda_i\}$ of $\tilde{L}$ are normalized to $q_i = \lambda_i / \sum_j \lambda_j$ and collapsed into spectral entropy $H_{\text{spectral}} = -\frac{1}{\log M}\sum_i q_i \log q_i \in [0,1]$. Low entropy signals one dominant event stage (simple query → Fast path); high entropy signals multiple independent stages (complex query → Slow path with threshold $\theta = 0.45$).

Dual-Path Localization. Fast path: the LVLM directly predicts $(t_s, t_e)$ from keyframes. Slow path (Temporal Markup Prompting, TMP): (1) Global temporal reasoning — model identifies key actions and their temporal/causal relationships; (2) Per-frame role annotation — each keyframe receives a monotonically non-decreasing label from $\{\text{before, start, during, end, after}\}$; (3) Boundary extraction — $t_s$ from the first start-labeled frame, $t_e$ from the last end-labeled frame.

Key Contributions¶

Query-conditioned DPP kernel that jointly encodes frame-query relevance and inter-frame diversity within a single greedy selection objective.
Dual-use of spectral entropy: the same DPP decomposition that selects keyframes also provides a principled, parameter-free difficulty indicator without additional computation.
Temporal Markup Prompting (TMP): a structured three-step CoT procedure with monotonicity constraints on temporal role labels, outperforming generic "think step by step" prompting.
Difficulty-adaptive routing framework applicable to any LVLM backbone, requiring no task-specific fine-tuning.
Adaptive frame budget selection that automatically scales keyframe count per (video, query) pair.

Results¶

Charades-STA (IID): +3.24 mIoU over prior best zero-shot method TAG.
ActivityNet Captions (IID): +3.34 mIoU over TAG.
OOD and compositional splits: gains of up to 3.4–3.5 mIoU over the strongest baseline across temporal-location-bias, split, and vocabulary OOD settings, and cross-dataset transfer.
Frame efficiency: >7× fewer frames than TFVTG while outperforming it.
Reasoning gap: DART reduces the simple-vs.-complex mIoU gap by nearly half compared to feature-matching methods (evaluated on 100+100 stratified queries from ActivityNet Captions val_2).

Limitations¶

Backbone is fixed to LLaVA-1.6-7B; performance sensitivity to LVLM choice is not analyzed.
Five hyperparameters ($\sigma_v, \sigma_t, \tau, \varepsilon, \theta$) are tuned on the Charades-STA validation split; cross-dataset robustness of these values is not ablated.
Spectral entropy threshold $\theta$ is a hard decision boundary; borderline queries may be misrouted.
Evaluation is limited to two benchmarks (Charades-STA, ActivityNet Captions); generalization to dense-event or egocentric video domains is untested.
Slow path TMP increases per-query latency for complex queries; no wall-clock profiling is reported.

Relevance to Vision-Language Models¶

DART demonstrates a principled strategy for eliciting structured temporal reasoning from frozen LVLMs without any fine-tuning, directly relevant to researchers studying zero-shot VLM capabilities for video understanding. The DPP-based keyframe selection addresses the perennial challenge of reducing visual token load while preserving event coverage—a general bottleneck for long-video VLM inference. The difficulty-adaptive routing pattern (spectral signal → conditional CoT activation) is a transferable design principle for efficient VLM inference across tasks where query complexity varies. The failure analysis quantifying the "reasoning gap" provides a useful diagnostic framework for evaluating VLM temporal reasoning deficits.