Skip to content

Provable Ordering and Continuity in Vision-Language Pretraining for Generalizable Embodied Agents

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AcTOL (Action Temporal Coherence Learning) addresses erroneous vision-language alignment in embodied agent pretraining caused by noisy goal-reaching assumptions in egocentric human action videos. It replaces rigid future-frame alignment with two theoretically grounded constraints—a Vision-Language Ordering (VLO) loss and a Brownian bridge continuity regularizer—that jointly enforce ordered and smooth VL representations without specifying goal frames. Pretrained features transfer to language-conditioned robot manipulation with strong data efficiency and robustness to linguistic variation.

Problem

Existing VL pretraining for embodied agents (R3M, LIV, DecisionNCE) uses time contrastive learning that aligns language with future or final frames under a goal-reaching assumption. Egocentric action videos (e.g., EPIC-KITCHENS-100) violate this assumption: actions may terminate early or contain irrelevant post-action content, causing the encoder to learn incorrect VL associations. No prior method enforces temporal ordering and smooth continuity of VL alignment without specifying explicit goal frames.

Method

AcTOL is initialized from CLIP (ResNet-50 backbone) and further pretrained on EPIC-KITCHENS-100 with two objectives:

Vision-Language Ordering (VLO) loss. For an anchor frame \(o_i\) and a nearby frame \(o_j\), the semantic alignment score \(R(v_i, v_j, l) = -\|\text{sim}(v_i, l) - \text{sim}(v_j, l)\|^2\) measures the VL-similarity difference. The negative set \(N_{i,j}\) contains frames temporally more distant from \(o_i\) than \(o_j\). An InfoNCE-style loss enforces that closer frame pairs exhibit smaller VL-similarity differences than more distant pairs—capturing temporal ordering without requiring any explicit positive goal frame.

Brownian Bridge (BB) loss. For a sampled interval \([n(i), n(j)]\), intermediate frame embeddings are regularized to follow the Brownian bridge distribution: mean linearly interpolating between endpoint embeddings, variance peaking at mid-interval. The loss \(L_\text{BB} = \frac{1}{T}\sum_t \frac{1}{2\text{Var}[B(t)]}\|v_t - \mathbb{E}[B(t)]\|^2\) enforces local visual continuity.

The combined loss is \(L_\text{AcTOL} = L_\text{VLO} + \lambda L_\text{BB}\) with \(\lambda{=}0.1\). At test time, encoders are frozen and a lightweight MLP policy is trained via behavior cloning on robot demonstrations.

Key Contributions

  • VLO loss that provably (Theorem 1) guarantees representations satisfy a temporal ordering property without goal-frame selection.
  • Brownian bridge continuity constraint that provably (Theorem 2) bounds VL alignment change for temporally close frames.
  • Theorem 3: robustness to language perturbations bounded by \(2C\delta_l\) under Lipschitz similarity.
  • End-to-end pretraining framework that transfers to language-conditioned imitation learning on both simulated and real robots with substantially fewer demonstrations than prior work.
  • Dense VL reward generation that correctly localizes action boundaries in multi-action video clips.

Results

Franka Kitchen (5 tasks, success rate over 50 roll-outs): - 5 demos: AcTOL 42.60% vs. best baseline (AcTOL w/o BB) 32.80% — +48.95% vs. strongest baseline (DecisionNCE 25.33%) - 15 demos: AcTOL 61.80% vs. DecisionNCE 43.20% (+43.06%) - 25 demos: AcTOL 64.60% vs. DecisionNCE 50.87% (+24.40%)

Metaworld (5 tasks): - 5/15/25 demos: AcTOL 53.81/74.13/81.13% vs. best baselines 50.29/70.83/73.33% (+14–16%)

Visual distribution shifts (Franka Kitchen, 15 demos): - AcTOL 43.2/32.8/9.2/4.4/38.4% vs. DecisionNCE 27.2/25.6/4.8/0.0/8.8% across S1–S5 shift conditions

Real robot (Unitree D1, 10 trials each): - Pick Cup: AcTOL 50% vs. DecisionNCE 20% - Open drawer: AcTOL 80% vs. DecisionNCE 40% - Close drawer: AcTOL 90% vs. DecisionNCE 60%

Fine-tuning with 25 robot demos (Franka Kitchen, 15-demo LCBC eval): frozen 61.8% → fine-tuned 86.4%

Linguistic robustness (Franka Kitchen): AcTOL success rate unchanged under instruction paraphrasing; LIV drops 11.9%, DecisionNCE drops 2.7%.

Limitations

  • Temporal ordering inductive bias breaks down for repetitive, cyclic, or ambiguous action sequences where monotone progression toward a goal does not hold.
  • Evaluated only on manipulation tasks; generalization to navigation or other embodied settings is untested.
  • Backbone limited to CLIP ResNet-50; scaling to transformer-based or larger vision encoders not explored.
  • Pretraining dataset (EPIC-KITCHENS-100) is kitchen-domain-specific; domain gap to diverse robot environments persists even with fine-tuning.

Relevance to Vision-Language Models

AcTOL directly concerns how VLMs can be adapted for temporally grounded, embodied settings, addressing a core failure mode of CLIP-based pretraining (misaligned goal frames) that is highly relevant to VLM practitioners extending foundation models to robotic or video understanding tasks. The Brownian bridge regularizer is a principled mechanism for imposing smooth temporal structure on frozen VLM representations—a technique applicable beyond robotics to any video-VLM alignment problem. The theoretical guarantees on ordering, continuity, and linguistic robustness provide formal grounding for empirical phenomena observed across VLM transfer learning literature. The work sits at the intersection of CLIP-style contrastive pretraining and embodied AI, making it directly actionable for researchers building VLM-based policies or reward models.