Skip to content

TRoVe: Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TRoVe is an automated pipeline for discovering static feature biases in temporal VLMs—biases where models exploit unchanging scene elements (backgrounds, objects, attributes) as shortcuts instead of analyzing true dynamic visual changes. It combines an error contribution score and a static bias score to rank candidate features extracted via clustering, then validates that knowing these biases enables test-time mitigation without retraining. Evaluated against 101 synthetic VLMs and 7 real-world models, TRoVe outperforms all baselines by 28.6%.

Problem

Temporal VLMs performing tasks like activity recognition or disease progression classification can latch onto static image features (e.g., trees, babies, bilateral opacities) rather than dynamic visual changes across a sequence. Prior methods for discovering such systematic failure modes (Domino, George, Distilling Failures) were designed for single-image settings and fail when the error-inducing feature appears in only a subset of a multi-image sequence. Additionally, no quantitative evaluation framework with ground-truth bias labels existed for the temporal setting.

Method

TRoVe operates on a pretrained VLM F and a labeled validation dataset DV in two steps:

Step 1 — Candidate feature extraction: Each image in every sequence is embedded by feeding a replicated static sequence (image Iᵢ repeated nᵢ times) into the VLM's vision encoder, stripping temporal variation. Embeddings are clustered via spherical K-means with cosine distance; the optimal cluster count is chosen by maximizing the Silhouette score. Each cluster C represents a recurring static visual pattern.

Step 2 — Scoring: For each cluster C and class label y, two scores are computed: - Error Contribution Score (ECS): acc^y_¬C − acc^y_C — accuracy drop on class y when the feature is present vs. absent. - Static Bias Score (SBS): Mean softmax confidence (at the predicted wrong label) when feeding static repetitions of misclassified images from C to F, with temperature-scaled calibration. High SBS indicates the model relies on the static feature even without temporal context.

The TRoVe score = ECS + SBS ranks clusters; top-ranked clusters are returned as discovered biases paired with affected class labels.

Mitigation: After discovery, CoOp prompt tuning is applied to sequences containing identified clusters—using only frozen VLM parameters—to improve test-time accuracy on bias-affected classes.

Evaluation framework: 101 synthetic temporal VLMs trained on circle-motion sequences with injected background, object, or attribute biases; ground-truth bias labels allow Precision@K and R-Precision evaluation.

Key Contributions

  • TRoVe: the first automated method specifically designed to discover static feature biases in temporal (multi-image sequence) VLMs.
  • A two-part scoring function (ECS + SBS) that jointly measures error contribution and model reliance on static cues.
  • A synthetic evaluation framework of 101 VLMs with known ground-truth biases across three feature types (background, object, attribute) and varying sequence lengths/prevalences.
  • Empirical demonstration that TRoVe-discovered biases can be used for training-free test-time mitigation via learned prompts (CoOp), yielding up to 111% accuracy improvement on affected class labels.
  • A comparative finding that non-temporal VLMs (CLIP variants) exhibit significantly more static biases (avg. 134.5/model) than temporal VLMs (avg. 84.5/model) on activity recognition.

Results

  • Synthetic benchmark (101 VLMs, Precision@10/25/100, R-Precision):
  • TRoVe achieves P@10 = 100% on background and attribute categories, 97.8% on object; closest baseline (Confidence) reaches 61.0%, 97.8%, and 64.6% respectively.
  • Overall 28.6% improvement over the closest baseline (Confidence method).
  • Real-world activity recognition (Kinetics400, Accuracy@5):
  • VideoCLIP-XL: Label ỹ accuracy 51.7% → 94.4% (+82.6 pp); Overall 82.2% → 86.7%.
  • ViCLIP-B: Label ỹ 45.3% → 95.8%; Overall 73.4% → 77.7%.
  • ViCLIP-L: Label ỹ 71.4% → 96.9%; Overall 77.1% → 80.7%.
  • Real-world medical imaging (MS-CXR-T, BioViL-T):
  • Accuracy on class "improving" drops 16.7 pp when severe-pneumonia markers are present, validating discovered bias.
  • Non-temporal baselines (Domino, George, Distilling Failures) perform near-random when naively extended to temporal settings.

Limitations

  • Requires a labeled validation dataset with class annotations; not applicable in purely unsupervised temporal settings.
  • Evaluation framework uses controlled synthetic data (moving circles); generalization to complex real-world diversity is validated qualitatively rather than with ground-truth annotations.
  • Mitigation strategy (CoOp prompt tuning) is demonstrated only on contrastive VLMs; applicability to generative or instruction-tuned VLMs is untested.
  • The number of image clusters K is selected via Silhouette score on the validation set, which may not generalize across all dataset distributions.
  • Only classification tasks are considered; regression or generation-based temporal tasks are out of scope.
  • Real-world bias validation relies on pseudo-labels from object detectors and entity extractors, not manual ground-truth annotations.

Relevance to Vision-Language Models

TRoVe addresses a fundamental reliability concern for VLMs applied to temporal tasks: that high aggregate performance can coexist with systematic shortcut learning tied to static visual cues, masking brittle failure modes. The method is directly applicable to any contrastive temporal VLM (VideoCLIP, ViCLIP, BioViL-T) and contributes a formal, scalable alternative to GradCAM-style manual auditing. For researchers studying VLM robustness, compositional reasoning, and temporal understanding, TRoVe establishes both a diagnostic tool and a quantitative evaluation protocol that fills a gap left by non-temporal bias-discovery literature. The medical imaging results also make it directly relevant to safety-critical VLM deployment scenarios.