RoboTTT: Context Scaling for Robot Policies¶

🕒 Published (v1): 2026-07-16 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

RoboTTT integrates Test-Time Training (TTT) into Vision-Language-Action (VLA) robot foundation models, scaling visuomotor context to 8K timesteps—three orders of magnitude beyond current robot foundation models—without growing inference latency. Fast weights, updated by gradient descent during both training and inference, compress long histories into parameter space, enabling one-shot imitation from human video, on-the-fly policy improvement, and robust long-horizon task completion.

Problem¶

State-of-the-art robot foundation models operate with single-step or short-history visuomotor context, preventing capabilities that require long temporal awareness: recognizing which assembly stage has been completed, recovering from failures with reference to prior state, and imitating from in-context human demonstrations. Naively concatenating past observations introduces spurious correlations and out-of-distribution issues at inference; standard recurrent models lack sufficient expressive capacity for dense, repetitive robot streams spanning thousands of timesteps.

Method¶

RoboTTT augments a pretrained VLA backbone (GR00T N1.7) with TTT layers inserted after each of the 16 DiT action-head attention layers. The TTT mechanism maintains fast weights \(W_t\) (a two-layer MLP per layer) updated at every timestep via gradient descent on an associative loss:

\[W_t \leftarrow W_{t-1} - \eta \nabla_W \mathcal{L}_\text{FW}(f_{W_{t-1}}(K_t), V_t)\]

and retrieves context via \(O_t = f_{W_t}(Q_t)\). This keeps inference cost constant in context length. A learned \(\tanh(\alpha)\) gate (initialized near zero) preserves pretrained capabilities.

Training recipe combines two techniques: - Sequence action forcing: independently samples the flow-matching noise level \(\tau_t\) per action chunk, stabilizing sequence-level diffusion training. - Truncated BPTT (TBPTT): segments the long sequence for gradient computation, carrying fast weights across boundaries but detaching their gradients, so GPU memory scales with segment length rather than total context.

Context utilization is enabled by masking the flow-matching loss on selected timesteps to create "pure context" steps: - One-shot imitation: human video sequences update fast weights only (loss masked); robot trajectory conditioned on the resulting fast weights provides action supervision. - DAgger Distillation: suboptimal robot actions update fast weights (loss masked); human correction actions provide the flow-matching target, distilling the failure→correction mapping into fast weight dynamics.

Key Contributions¶

First demonstration that pretraining context length is a scaling axis for robot foundation models, with steady closed-loop performance gains from 128 to 8K timesteps.
TTT-based architecture that scales visuomotor context to 8K timesteps with constant inference cost, instantiated on GR00T N1.7.
Sequence action forcing: per-chunk independent noise sampling that stabilizes sequence-level flow-matching training.
DAgger Distillation: asymmetric use of failure history (fast weight context) and corrections (imitation targets) as an Algorithm Distillation instantiation for robotics.
One-shot in-context imitation from human video demonstrations via masked-loss context conditioning.

Results¶

Overall task completion: RoboTTT averages 79% task completion score vs. 42% for GR00T N1.7 (single-step) — 87% relative improvement — and vs. 56% for GDN (best baseline) — 41% relative improvement.
Gear Bot (5-min, 10-stage assembly): RoboTTT achieves 2/10 full successes; all baselines achieve 0/10.
Pup Go Car: RoboTTT 9/20 full successes vs. GR00T N1.7 3/20, GR00T N1.7 Hist. 0/20, GDN 3/20.
Circuit: RoboTTT 13/20 vs. GR00T N1.7 3/20, GR00T N1.7 Hist. 8/20, GDN 8/20.
Context length scaling: RoboTTT-8K reaches 71.5% average completion, 63% higher than the same model at 1K context (43.9%) and 57% higher than GR00T N1.7 Hist. (45.6%); no saturation observed. GDN shows no scaling trend.
Perturbation robustness: RoboTTT succeeds in 83% of perturbed trials vs. 53% for the best short-context baseline.
On-the-fly improvement (DAgger Distillation): 36% better performance than the same model without this capability.
One-shot imitation: succeeds in 6/10 trials; GDN and GR00T N1.7 fail entirely.

Limitations¶

Post-training uses only 1K-timestep context even for RoboTTT-8K, meaning the full pretraining context benefit is not exploited during task fine-tuning.
Below 1K pretraining context, rollout horizons exceed training context, causing fast weights to update beyond their trained window and positional embeddings to extend to unseen positions.
DAgger Distillation requires collecting interleaved human-correction rollout data, adding data collection overhead.
Evaluation is limited to bimanual tabletop assembly tasks on a single robot platform (YAM); generalization to other morphologies and task domains is untested.
The text is truncated before the full one-shot imitation quantitative results table are shown.

Relevance to Vision-Language Models¶

RoboTTT directly extends VLA models—a major class of VLMs applied to robotics—by replacing or augmenting the short-context attention mechanism with TTT-based long-context conditioning, demonstrating that context scaling laws observed in pure language models transfer to embodied visuomotor settings. The architecture is backbone-agnostic and compatible with any VLA that uses a DiT action head, making it immediately applicable to the growing family of large-scale VLA policies. The one-shot imitation capability via in-context video conditioning is a direct robotics analogue of few-shot in-context learning in LLMs, validating that long-context VLMs can be repurposed as policy scaffolds without task-specific fine-tuning. For VLM researchers, this work establishes context length as an empirically validated scaling axis beyond language, with measurable and monotonic gains across orders of magnitude.