Skip to content

Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detection

🕒 Published (v1): 2025-11-25 10:38 UTC · Source: Arxiv · Venue: CVPR · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current state-of-the-art video LMMs (GPT-5.2, Gemini-3-Pro, Qwen-2.5VL, GPT-4o, Gemini-2.5-Flash) can reliably name actions and objects in hand-object interaction videos but fail near chance-level at pinpointing the precise frame where physical contact or release occurs. The paper introduces CRID, a 20K+ human-annotated dataset of contact/release events from SSv2, and frames the failure as a System 1 vs. System 2 dissociation: models pattern-match semantics but lack the causal physical grounding needed for fine-grained spatiotemporal reasoning.

Problem

Video LMMs achieve strong action recognition by leveraging statistical co-occurrence of visual tokens (hands near objects → "picking up"), but whether this reflects genuine physical understanding or superficial pattern matching is unknown. No prior benchmark specifically targets the precise spatiotemporal moment of physical contact/release—the primitive underlying all manipulation reasoning.

Method

The authors construct CRID: 10,130 SSv2 videos annotated by 250 AMTurk workers for 24,222 contact/release events, including event type (contact/release), agent type (hand/object/surface), and spatiotemporal location (frame number + image coordinates; ICC=0.95 for frame, IoU=0.57 for 120×120 bounding box). From a 33-video subset they extract 99 temporally isolated 10-frame clips (one event per clip, always at 10-frame length so chance = 10%) plus matched no-event control clips. Five SoTA LMMs are evaluated under three ICL regimes (zero-shot, one-shot, two-shot) crossed with two prompting modifiers—Grounding (model first names objects/describes scene) and Reasoning (chain-of-thought)—yielding 12 experimental conditions. Spatial localization is evaluated as IoU between predicted and ground-truth 120×120 bounding boxes. Human performance on identical clips provides an upper bound.

Key Contributions

  • CRID dataset: 20K+ frame-level and spatial annotations of contact/release events on 10K SSv2 videos; publicly released.
  • Controlled evaluation protocol: 99 single-event 10-frame clips + matched no-event controls, evaluated under ZS/OS/TS × Grounding × Reasoning conditions.
  • Empirical grounding gap: Quantified dissociation between high semantic accuracy (>80% Top-5 action, >70% object naming) and near-chance physical event localization (<17% exact frame, mean IoU ≈0.086–0.147).
  • Null results for prompt engineering: Chain-of-thought and grounding prompts provide no consistent improvement and can decrease performance.
  • Human baseline: Establishes human-level targets (43% exact, 81% 1-off frame; IoU 0.71).

Results

  • Frame detection (ZS, exact): All models ≈10% (chance); best is Gemini-3-Pro at 42.9% with 1-frame tolerance.
  • Frame detection vs. humans: Humans score 43.0% exact / 81.4% at 1-frame off vs. best model's ~20% exact / ~49% at 1-frame off (TS regime).
  • Spatial localization (IoU): Gemini-3-Pro 0.147, GPT-5.2 0.086 vs. human 0.71.
  • No-event control: All models except Qwen always hallucinate an event frame; Qwen correctly abstains 18.2% of the time.
  • Action/object recognition: >80% Top-5 action accuracy; >70% object naming across models—confirming semantic capability is intact.
  • Reasoning condition: No consistent improvement; sometimes hurts (e.g., GPT-5.2 TS drops from 45.2% to 39.9% at 1-frame off with Reasoning).
  • Grounding condition: Similarly null or negative effect.
  • Few-shot ICL: Minor improvement (ZS→TS), but remains near chance for exact detection.

Limitations

  • Evaluation uses only 99 clips from 33 videos—small experimental set even if CRID itself is large.
  • Single annotator per video in CRID (only a random subset had ICC measured); spatial annotations have moderate inter-rater agreement (ICC 0.39–0.73 for x/y).
  • Clips are exactly 10 frames, making chance level a fixed 10% and potentially constraining ecological validity.
  • No fine-tuning experiments; all evaluations are inference-only, leaving open whether CRID annotations could actually close the gap.
  • Physical contact primitives in SSv2 are hand-centric; generalization to object-object or tool-use contexts is untested at this evaluation scale.

Relevance to Agentic AI / LLM Agents

An agent that plans or executes physical manipulation—robot control, embodied AI, tool-use reasoning—must detect when and where contact occurs to correctly model state transitions. This paper provides direct evidence that even the most capable multimodal LLMs (including GPT-5.2 and Gemini-3-Pro) cannot reliably perceive these causal physical primitives, implying that LMM-based agents relying on visual feedback for manipulation will have a fundamental grounding deficit. The System 1/System 2 framing is particularly relevant to agentic architectures: an agent may narrate a plausible plan while being blind to the actual physical state changes that confirm or deny execution. The CRID dataset and evaluation protocol offer a concrete diagnostic benchmark for any agentic video-grounding capability.