PolicyTrim: Boosting Intrinsic Policy Efficiency of Vision-Language-Action Models

🕒 Published (v1): 2026-06-21 14:54 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; intrinsic policy efficiency for VLA deployment; directly buildable

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PolicyTrim is a two-stage RL post-training framework that improves the intrinsic policy efficiency of VLA models—distinct from per-step compute efficiency—by extending reliable action chunk execution lengths and eliminating redundant physical steps. Applied to π0.5, OpenVLA-OFT, and GR00T without architectural changes or extra demonstrations, it delivers up to 5.83× end-to-end deployment speedup with no loss in task success rate.

Problem

Existing VLA efficiency work targets per-step inference latency (token pruning, quantization, KV-cache reuse), but leaves policy efficiency untouched: (1) action chunk tail predictions degrade in quality, forcing conservative truncation of the execution window \(h = \lfloor \gamma H \rfloor\) with \(\gamma \ll 1\), and (2) imitation-learned policies produce redundant physical steps even when shorter paths are reachable. Total execution time is jointly determined by per-step latency and the number of inference calls; optimizing only the former is fundamentally bounded when the call count is excessive.

Method

PolicyTrim is a two-stage GRPO-based post-training framework:

Stage 1 — Reliable Action Chunk Extension. Within each sampled group of \(G\) trajectories, each trajectory \(\tau_i\) is assigned a distinct acceptance ratio \(\gamma_i \in \Gamma = \{\gamma_1, \ldots, \gamma_M\}\), yielding execution windows \(h_i = \lfloor \gamma_i H \rfloor\). This turns one group rollout into a parallel reliability sweep across short-to-long chunk positions. The integrated reward is: $\(R_{\text{ext}}(\tau_i) = \mathcal{I}_{\text{succ}}^{(i)} \cdot (R_{\text{succ}}(\tau_i) + \beta \cdot \gamma_i)\)$ where the horizon bonus \(\beta \cdot \gamma_i\) is gated on group-level success to prevent pushing toward longer but unreliable chunks when the task is too hard. Group-relative advantages are computed via GRPO normalization, and the policy is updated with a clipped surrogate + KL penalty to the pre-training reference.

Stage 2 — Redundancy-Aware Step Reduction. Initialized from Stage 1, trajectories completing the task in \(S(\tau_i)\) steps receive a step-saving reward: $\(R_{\text{step}}(\tau_i) = \frac{\max(0, S_{\text{base}} - S(\tau_i))}{S_{\text{base}}}\)$ A group-anchored stability penalty prevents collapse onto irreproducible shortcuts: $\(P_{\text{stab}}(\tau_i) = \lambda_{\text{stab}} \cdot \tanh\!\left(\frac{|S(\tau_i) - \mu_{\text{group}}|}{\max(\sigma_{\text{group}}, \sigma_{\text{floor}})}\right)\)$ The joint reward \(R_{\text{eff}} = \mathcal{I}_{\text{succ}}^{(i)} \cdot (R_{\text{succ}} + R_{\text{step}} - P_{\text{stab}})\) is again optimized via GRPO. The two stages are intentionally sequential: chunk extension first widens the reliable prediction horizon; step reduction then compresses the physical trajectory on top of that wider horizon.

Key Contributions

• Identifies policy efficiency (chunk utilization + physical step count) as an orthogonal and overlooked deployment bottleneck distinct from computational efficiency.
• Dynamic Execution Horizon Exploration: assigns distinct acceptance ratios per trajectory within a group to sweep reliability boundaries without extra rollouts.
• Reliable Horizon Reward with group-success gating, preventing horizon bias when no trajectory succeeds.
• Redundancy-Aware Step Reduction with group-anchored \(\tanh\) stability penalty and a floor parameter \(\sigma_{\text{floor}}\) to avoid over-penalization at convergence.
• Demonstrated cross-architecture generalization on three VLA paradigms (diffusion π0.5/GR00T, parallel-decode OpenVLA-OFT) across three simulation benchmarks and real-robot transfer.

Results

• Step reduction: Physical steps cut to 51.4% of baseline on π0.5 LIBERO-Object (125.0 → 64.3 steps); consistent reductions across all benchmarks and models.
• Chunk utilization: 3× improvement in average action chunk execution length (e.g., π0.5 LIBERO-Spatial: \(h_{\text{chunk}}\) 5 → 15).
• End-to-end speedup: Up to 5.83× (π0.5 LIBERO-Object); 5.43× on LIBERO-Spatial; 2.91× on LIBERO-Long; 2.36× on ManiSkill.
• Success rate: Maintained within noise across all settings (e.g., π0.5 LIBERO-Object: 99.1% → 98.5%; OpenVLA-OFT LIBERO-Spatial: 98.6% → 98.8%).
• OpenVLA-OFT caveat: Parallel bidirectional decoding causes severe accuracy degradation under any chunk extension; only Stage 2 (step reduction) is applied; speedup still 1.79× on LIBERO-Spatial.
• ManiSkill: π0.5 steps 45.2 → 38.3 (2.36× speedup); success rate 88.1% → 89.8%.
• Meta-World MT50: π0.5 steps 66.3 → 52.6 (2.52× speedup) at near-identical SR.

Limitations

• Stage 1 (chunk extension) is architecture-dependent: parallel-decode models like OpenVLA-OFT cannot benefit from it due to degradation under longer chunks.
• The two-stage sequential design assumes chunk extension is a prerequisite for effective step reduction; joint optimization is explicitly avoided but not ablated as a baseline.
• \(S_{\text{base}}\) is set to ~1.3× the initial average successful step count—a heuristic that may require tuning per task domain.
• Real-world evaluation is limited to three tabletop tasks on a single arm platform (Agilex Piper); generalization to dexterous or contact-rich manipulation is undemonstrated.
• No wall-clock timing breakdown distinguishing inference latency savings from step-count savings, making it hard to predict gains when combined with complementary compute-centric methods in novel settings.

Relevance to Vision-Language Models

PolicyTrim addresses a critical gap for VLMs deployed as action policies: while the VLM backbone drives much of VLA capability, its inference cost is multiplied by the number of planning cycles, so policy-level inefficiency compounds backbone expense. The work shows that GRPO—originally developed for reasoning VLMs—can be repurposed to reshape the behavioral distribution of VLAs toward efficiency objectives, not just correctness. This establishes a general post-training recipe applicable on top of any frozen or fine-tuned VLM backbone without retraining the vision-language components. For researchers tracking VLMs, it highlights that the action chunking interface between VLM representations and robot control is itself a learnable efficiency lever, separable from model architecture.