Real-Time Execution of Action Chunking Flow Policies¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Real-Time Chunking (RTC) is an inference-time algorithm that enables smooth, asynchronous execution of action-chunking diffusion/flow VLAs without retraining. It frames the chunk-transition problem as inpainting: freezing actions already committed to by inference delay and using pseudoinverse guidance with soft masking to generate the remainder. RTC achieves 20% faster real-world task execution and remains fully robust to >300ms inference delays where competing methods fail entirely.

Problem¶

State-of-the-art VLAs (e.g., π0, 3B params at 46ms prefill alone for 50Hz control) cannot produce action chunks within a single controller timestep. Naive synchronous inference causes visible pauses; naive asynchronous inference causes out-of-distribution, jerky mode-jumps at chunk boundaries because consecutive chunks may commit to incompatible strategies. Temporal ensembling averages multi-modal action distributions, producing invalid intermediate actions.

Method¶

RTC poses chunk-to-chunk continuity as a flow-matching inpainting problem. At inference start, the d actions guaranteed to execute before the new chunk is ready are "frozen" to their previously planned values. The remaining H − d actions are inpainted using training-free pseudoinverse guidance (ΠGDM): at each denoising step, a gradient correction nudges the partially denoised trajectory toward a soft-masked target derived from the previous chunk. The soft mask assigns weight 1 to the frozen d actions, exponentially decaying weights to the intermediate H − d − s overlap region, and weight 0 to the final s freshly generated actions. A clipped guidance weight β prevents instability with the small number of denoising steps typical in control (n=5). The inference loop runs in a background thread; execution horizon s adapts per-chunk based on a rolling buffer of observed delays.

Key Contributions¶

Real-Time Chunking (RTC): a training-free, inference-time inpainting algorithm compatible with any diffusion- or flow-based VLA.
Soft masking: exponentially decaying guidance weights over the overlapping chunk region, extending continuity beyond the hard-frozen prefix.
Kinetix dynamic benchmark: 12 highly dynamic simulation tasks (throw, catch, balance, locomotion) where quasi-static benchmarks are insufficient.
Real-world evaluation: 6 bimanual manipulation tasks with π0.5, 480 total episodes (~28 hours), including injected latency sweeps (+0/+100/+200ms).

Results¶

Simulation (Kinetix, 12 tasks, 2048 rollouts/point): RTC outperforms all baselines (naive async, temporal ensembling, BID) across inference delays d=0–4; gap widens with delay. Soft masking improves over hard masking at low d. RTC uniquely shows strictly increasing solve rate as execution horizon decreases.
Real world (π0.5, 6 tasks):
RTC achieves 20% faster task completion vs. synchronous inference (measured in controller steps, pauses excluded).
RTC shows no degradation in average throughput across +0/+100/+200ms injected delay; synchronous degrades linearly.
Both TE variants (sparse and dense) trigger robot protective stop at +100ms and +200ms due to high-acceleration oscillations; RTC runs robustly.
On light candle (precision task, no retry): RTC shows the largest advantage in final success score.
RTC latency: 97ms vs. 76ms for synchronous baselines (21ms overhead for the guided inference).
BID (batch size 16) has 2.3× the latency of RTC on π0.5.

Limitations¶

Adds computational overhead vs. direct sampling: 97ms vs. 76ms per chunk for π0.5 (≈28% overhead from the vector-Jacobian product at each denoising step).
Applicable only to diffusion- and flow-based policies; not directly usable with autoregressive VLAs.
Real-world evaluation limited to arm manipulation; legged locomotion (present in simulation benchmark) not tested on hardware.
Guidance quality degrades when d is very small (weak frozen-prefix signal), partially mitigated by soft masking but not fully resolved.

Relevance to Vision-Language Models¶

VLAs are the dominant paradigm for generalizable robot manipulation, built on VLM backbones (π0, RT-2, OpenVLA), and their high inference latency is a fundamental deployment bottleneck. RTC directly addresses this bottleneck at inference time, requiring no retraining of the underlying VLM backbone, making it immediately applicable to any deployed flow-based VLA. The inpainting framing connects diffusion/flow guidance literature (ΠGDM, Diffuser) to the control setting in a novel way. As VLAs scale in parameter count and are pushed to remote/cloud inference, latency will only increase, making latency-robust inference algorithms like RTC increasingly critical for the field.