Skip to content

AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models

🕒 Published (v1): 2026-07-02 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AnyGroundBench is the first domain-adaptation benchmark for Spatio-Temporal Video Grounding (STVG) in five specialized fields (animal, industry, sports, surgery, public security), shifting evaluation from zero-shot general-domain testing to few-shot domain adaptation. It exposes severe, previously unreported failures: all 15 evaluated VLMs—including the best proprietary models—collapse on specialized-domain STVG, with spatial localization identified as the primary bottleneck. In-Context Learning (ICL) as an adaptation strategy is unreliable and often degrades performance.

Problem

Existing STVG benchmarks (VidSTG, HC-STVG, OmniGround) are confined to general daily-life footage, and evaluation protocols measure only zero-shot performance. Real deployments—surgical instrument tracking, laboratory animal behavior, industrial assembly—require models to handle rare visual concepts and complex spatiotemporal dynamics absent from pretraining. No standardized benchmark exists to measure VLM adaptability to such specialized domains under practical (few-shot, compute-constrained) conditions.

Method

AnyGroundBench provides 2,040 videos and 3,522 query–annotation pairs across five domains, each with dedicated train/test splits enabling domain-adaptation evaluation. Three interconnected tasks are unified under the same VLM \(F_\theta\) invoked via task-specific system prompts \(p \in \{p_{\text{STVG}}, p_{\text{SVG}}, p_{\text{TVG}}\}\):

  • STVG: predict the full spatio-temporal tube \(\hat{\tau} = \{(t, \hat{b}_t)\}_{t=\hat{t}_s}^{\hat{t}_e}\)
  • SVG: given ground-truth temporal span, predict per-frame bounding boxes \(\{\hat{b}_t\}\)
  • TVG: predict only temporal boundaries \([\hat{t}_s, \hat{t}_e]\)

SVG and TVG decompose the joint task to isolate spatial vs. temporal failure modes. The benchmark is adaptation-method-agnostic: any operator \(g = A(F_\theta, \mathcal{T}^{\text{train}}_D)\) is directly comparable. The reference instantiation is \(m=2\)-shot ICL with top-\(m\) retrieval using cosine similarity weighted equally (0.5) over SentenceBERT text embeddings and InternVideo2 visual embeddings.

Annotation combines Grounding DINO (open-vocabulary detection) and SAM2 (tracking) for automated/semi-automated box labeling, followed by mandatory per-frame manual inspection and independent double-checking. Domain experts (medical doctors for mouse scratching, former players for American football) annotated newly captured datasets.

Key Contributions

  • First STVG benchmark targeting specialized domains with dedicated per-domain training subsets for adaptation evaluation
  • Two novel datasets: Mouse Scratching (expert-annotated, clinical fidelity) and American Football (domain-expert annotators for tactical play sequences)
  • Decomposition of STVG into isolated SVG and TVG subtasks to pinpoint which reasoning component fails
  • Evaluation of 15 VLMs (6 proprietary, 1 specialized open-source, 8 general open-source) under zero-shot and 2-shot ICL conditions
  • Quantitative characterization of ICL instability as a domain adaptation strategy for grounding

Results

All metrics at threshold 0.3 (vIoU@0.3 for STVG, tIoU@0.3 for TVG, sIoU@0.3 for SVG). Results reported as STVG / TVG / SVG:

  • Best proprietary model (Gemini-3.1-Pro, zero-shot): 16.5 / 37.5 / 70.7 on Animal; 7.69 / 21.8 / 41.4 on Industry; 4.16 / 32.8 / 26.1 on Sports; 22.8 / 69.4 / 52.0 on Surgery; 1.22 / 26.3 / 16.5 on Public Security
  • All open-source general-purpose models (e.g., InternVL3-14B, Qwen3.5-9B): STVG scores of 0.00 across nearly all domains; SVG scores also at 0.00 on Sports and Surgery
  • ICL is inconsistent: Gemini-3.1-Pro improves Industry SVG from 7.69 → 11.8 with +ICL, but degrades Animal STVG from 16.5 → 12.7; open-source models show near-zero gain or regression
  • TVG is more tractable than SVG: TVG scores reach 60–70% on Surgery (Gemini-3.1-Pro: 69.4), while STVG collapses due to spatial grounding failure
  • Average video statistics: 47.1 s mean duration, 7.51 s mean target segment (16% of video), 10.63% mean relative bounding box area

Limitations

  • ICL is the only adaptation method evaluated; PEFT and fine-tuning baselines are deferred to benchmark users, not measured in the paper
  • The benchmark text is truncated before full analysis of results across all threshold levels (\(@0.1\), \(@0.5\)) and all domains
  • 2-shot ICL may underrepresent models' full few-shot potential; 4-shot results shown only for Gemini-3.1-Pro
  • Newly captured datasets (Mouse Scratching, American Football) are single-institution/source, limiting distributional diversity within those subsets
  • No analysis of failure modes conditioned on query complexity, object size, or motion speed

Relevance to Vision-Language Models

AnyGroundBench directly stress-tests a core VLM capability—joint spatial and temporal perception from language—in the out-of-distribution regime that deployment demands, revealing that current architectures have a fundamental spatial reasoning bottleneck decoupled from their temporal capabilities. The task decomposition (SVG vs. TVG) is a methodologically clean diagnostic that future VLM designs can use to target spatial grounding explicitly. The benchmark's adaptation-agnostic protocol accommodates both prompt-based methods (ICL) applicable to closed-source VLMs and gradient-based methods (PEFT), making it a practical testbed as the field moves toward instruction-tunable and agentic VLMs. The finding that ICL frequently degrades spatial grounding is directly relevant to practitioners deploying VLMs in specialized domains via in-context demonstrations.