STVG-R1: Incentivizing Instance-Level Reasoning and Grounding in Videos via Reinforcement Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

RL reduces hallucination in spatial-temporal video grounding; ICLR 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

STVG-R1 tackles spatial-temporal video grounding (STVG) by replacing dense per-frame bounding-box regression with a compact object-ID identification problem: each video instance is overlaid with a unique numeric ID as a visual prompt, and the model simply predicts an ID plus a temporal interval. A GRPO-based reinforcement learning stage with decomposed spatial, temporal, and format rewards then fine-tunes Qwen2.5-VL-7B, surpassing the base model by +20.9% mIoU on HCSTVG-v2 and generalizing zero-shot to multi-object referring video segmentation (47.3% J&F on MeViS).

Problem

VLMs hallucinate in dense prediction tasks (e.g., STVG) because aligning natural-language descriptions with raw pixel coordinates is fundamentally difficult—models output timestamps beyond video duration or coordinates outside frame boundaries. Alignment-based fixes add large trainable modules requiring costly annotated data; decoder-based methods generate implicit segmentation tokens that limit generalization. Neither handles multi-object scenes robustly.

Method

Object-centric visual prompting. Each video frame \(I_t\) is augmented as \(\tilde{I}_t \triangleq I_t \oplus P_t\), where \(P_t = \{p^t_1,\ldots,p^t_{K_t}\}\) are numeric ID markers placed at instance centroids. Instance tracks are built by: (1) running YOLOv12x on frame 1 for candidate boxes; (2) propagating masks via SAM2.1-large; (3) re-detecting every 15 frames and registering new instances when IoU with all tracked objects falls below 0.4. Ground-truth target ID \(A\) is determined by majority voting over per-frame IoU argmaxes (Eq. 3). The VLM thus only needs to predict a scalar ID \(\imath\) and a temporal interval \([t_s, t_e]\), not per-frame coordinates.

STVG-R1 reward design. Three scalar rewards are summed: $\(R(o) = r_t(o) + r_s(o) + r_f(o)\)$ - \(r_t(o)\): temporal IoU between predicted \([t_s,t_e]\) and ground-truth \([t'_s,t'_e]\) (continuous, Eq. 4). - \(r_s(o) \in \{0,1\}\): 1 iff predicted ID \(\imath = \imath^*\) and \(\imath\) appears within \([t_s,t_e]\) (Eq. 5). - \(r_f(o) \in \{0,1\}\): enforces <think>…</think> / <answer>…</answer> output structure.

RL training. GRPO with \(n=8\) rollouts per query; advantage normalized within the group (Eq. 7); clipped policy ratio with KL penalty toward a frozen reference policy (Eq. 8). Trained for 1 epoch on merged HCSTVG-v1/v2 + VidSTG training splits using Qwen2.5-VL-7B as backbone (8×A100, lr \(10^{-6}\)).

Key Contributions

• Training-free object-centric visual prompting that converts per-frame coordinate regression to a compact instance-ID lookup, applicable plug-and-play to any VLM.
• STVG-R1: first GRPO-based RL framework specifically designed for spatial-temporal video grounding.
• Decomposed reward combining continuous temporal IoU, binary spatial consistency conditioned on the localized segment, and structural format compliance.
• Demonstrated zero-shot transfer to multi-object referring video object segmentation (MeViS) without any segmentation-specific training.

Results

• HCSTVG-v2 (val): STVG-R1 achieves mIoU 40.8% — a +20.9% gain over Qwen2.5-VL-7B (19.3%) and +6.8% over SpaceVLLM-7B (34.0%), and sets new SOTA across all four reported metrics.
• HCSTVG-v1 (test): STVG-R1 39.1% mIoU, matching or exceeding all VLP-specialized baselines (TubeDETR 36.4%, TA-STVG 40.2%) with far fewer inductive biases.
• ST-Align: STVG-R1 surpasses LLaVA-ST by +0.6% mVIoU on STVG and +13.1% mVIoU on video spatial grounding.
• Zero-shot visual prompting (no RL): vIoU@0.3 gains on HCSTVG-v1 of +3.6% (InternVL3-8B), +12.5% (Qwen2.5-VL-7B), +6.0% (Qwen2.5-VL-72B), +28.3% (Qwen3-VL-8B).
• MeViS (zero-shot): 47.3% J&F, SOTA on multi-object referring video object segmentation without any task-specific training.

Limitations

• Visual prompt overlays occlude fine-grained frame details, causing slight temporal localization regressions for Qwen2.5-VL models (−1.6 to −1.9% mTIoU on HCSTVG-v1).
• Memory budget forces aggressive down-sampling: a 30-second video at 2 FPS yields 60 frames each at roughly 96×96 pixels, potentially losing texture information.
• Global detection failure (target undetected in all frames) affects ~1% of samples; local misses require periodic re-detection and a post-hoc ID-repair step at inference.
• Depends on COCO-taxonomy detectors, introducing semantic label noise (though instance identities rather than class labels drive supervision).
• Spatial consistency reward is binary and cannot provide gradient signal on near-miss ID predictions.

Relevance to Vision-Language Models

This paper addresses one of the most practically consequential failure modes of VLMs—coordinate hallucination in dense spatiotemporal prediction—by cleverly decoupling object identity from coordinate regression via visual prompts, avoiding any modification to the model architecture. The successful application of RLVR (GRPO) to STVG extends the DeepSeek-R1/Video-R1 paradigm into structured spatiotemporal output spaces, demonstrating that verifiable multi-component rewards enable coherent chain-of-thought reasoning over object IDs and time intervals. The plug-and-play nature of the prompting strategy, shown to benefit four distinct VLMs in zero-shot settings, is immediately relevant to practitioners seeking to improve grounding without retraining. The zero-shot generalization to mask-level segmentation further suggests object-centric visual prompting as a broadly applicable interface between VLMs and spatial perception modules.