Skip to content

Patch Policy: Efficient Embodied Control via Dense Visual Representations

🕒 Published (v1): 2026-07-20 17:59 UTC · Source: Arxiv · link

Why this paper was selected

Yann LeCun co-author; dense ViT features for embodied control, underexplored direction

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PATCH POLICY is a lightweight robot policy architecture that ingests dense, frozen ViT patch tokens directly via a block-causal attention mask, bypassing the spatial compression bottleneck of global-pooled features. It achieves a 40% relative improvement over global-feature policies and outperforms fine-tuned OpenVLA-OFT by 18% using only ~0.7% of its parameters. The work argues that dense spatial representations—already embedded in off-the-shelf pretrained ViTs—are the key ingredient VLAs provide, obtainable without billion-parameter generative backbones.

Problem

Existing robot policies either compress visual observations into a single global token (CLS token or average pooling), losing fine-grained spatial detail critical for precise manipulation, or fine-tune billion-parameter VLMs as vision-language-action (VLA) models, incurring prohibitive training and inference costs. No efficient intermediate existed that exploited dense pretrained ViT patch features without the overhead of a full VLM backbone.

Method

PATCH POLICY is a modular transformer-based policy with two components:

Observation Trunk: Given image observation \(o_t \in \mathbb{R}^{C \times H \times W}\), a frozen pretrained ViT encoder produces \(P \times D\) patch features per frame. Over a context window of length \(T\), this yields a spatio-temporal tensor of shape \(T \times P \times D\). Goal images are channel-concatenated (\(T \times P \times 2D\)); goal vectors are appended to every token (\(T \times P \times (D+G)\)).

Policy Head with Block-Causal Attention: The \(T \times P\) token sequence receives learned 1D positional embeddings and is processed with a block-causal mask: full bidirectional attention within each frame (intra-frame), causal masking across frames (inter-frame). This preserves temporal causality while allowing spatial integration. The policy head (either VQ-BeT or Diffusion Policy) emits action chunks from the final patch token of each frame. Inference uses a rolling context window with receding-horizon control. The ViT encoder is always frozen; only the policy head trains.

Key Contributions

  • Block-causal attention mask enabling standard transformer policies to consume dense patch sequences without architectural overhaul.
  • Demonstration that frozen off-the-shelf ViT patch features (no encoder fine-tuning) suffice for robust robotic control, including real-world tasks with ~2 mm tolerances.
  • Systematic benchmark of five pretrained encoders (DINOv2, DINOv3, WebSSL, V-JEPA 2, SigLIP 2) as frozen policy backbones, finding WebSSL and DINOv2 best.
  • Evidence that spatial compression (pooling or learned convolution) monotonically degrades manipulation performance.
  • Efficiency analysis: VQ-BeT variant with DINOv2 runs at 10.99 ms inference latency and trains in 6.5 GPU-hours vs. 16 GPU-hours for OpenVLA-OFT.

Results

  • Simulated environments (Push-T, LIBERO Goal, BlockPush, Cube): WebSSL patch + Diffusion Policy scores 1.68/1.73 on BlockPush/Cube vs. 0.99/0.21 for WebSSL CLS + Diffusion Policy; outperforms OpenVLA-OFT (1.43/1.50) on the same tasks.
  • 40% relative improvement over state-of-the-art global-pooled (WebSSL CLS) baselines across the four simulated suites.
  • 18% improvement over fine-tuned OpenVLA-OFT across simulated benchmarks.
  • Real-world cable insertion (20 trials): DINOv2 patch + VQ-BeT achieves 0.70 full insertion vs. 0.60 (CLS), 0.35 (ACT/ResNet-18), 0.30 (OpenVLA-OFT).
  • Real-world pen collection: 0.85 (three pens) vs. 0.65 (CLS), 0.65 (ACT), 0.60 (OpenVLA-OFT).
  • Parameter count: 51.55M total / 29.49M trainable (DINOv2 VQ-BeT) vs. 7.61B for OpenVLA-OFT (~0.7% of parameters with ViT-S).
  • Patch compression ablation: reducing from 256 to 4 patches on Push-T drops success from 0.69 → 0.51.
  • SigLIP 2 underperforms across all tasks, attributed to language-image alignment sacrificing geometric features.

Limitations

  • Only frozen encoder regimes are evaluated; end-to-end fine-tuning for specialized domains is unexplored.
  • Dense patch sequences increase token length and training time; FlashAttention integration is left to future work.
  • Evaluated only under behavior cloning; extension to reinforcement learning (to surpass the expert-demonstration ceiling) is not addressed.
  • Inference latency scales with number of patches: switching DINOv2 CLS → DINOv2 patch doubles VQ-BeT latency (5.79 ms → 10.99 ms), though still within real-time requirements.

Relevance to Vision-Language Models

PATCH POLICY directly challenges the prevailing assumption that dense spatial features useful for embodied control require a full VLM backbone, showing that a lightweight policy consuming frozen ViT patch tokens can match or exceed billion-parameter VLAs like OpenVLA-OFT. The finding that SigLIP 2—a leading vision-language encoder—underperforms DINOv2 and WebSSL on manipulation tasks suggests that language-image alignment may trade off geometric representational quality, an important signal for VLM design and transfer. For researchers tracking VLMs, this paper provides a controlled encoder comparison (DINOv2, DINOv3, WebSSL, V-JEPA 2, SigLIP 2) that reveals which pretraining objectives produce spatially rich features transferable to non-linguistic downstream tasks. It reframes the VLA paradigm: the dense ViT features matter far more than the language grounding for precise manipulation, isolating the representational contribution of the vision tower from the rest of the VLM.