Skip to content

Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Models

🕒 Published (v1): 2026-07-04 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Distills tree search into frozen VLA action evaluation; novel adaptation without fine-tuning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLA models already contain successful behaviors in their output distributions but lack the ability to select them, a bottleneck SVA addresses by distilling Monte-Carlo tree search into a lightweight Q-value model that scores candidate actions at deployment. SVA keeps the VLA backbone entirely frozen, preserving generalist capability, while the learned evaluator selects the highest uncertainty-regularized Q-value action from N proposals. A 9B VLA augmented with SVA outperforms a 27B VLA by 7 points at 27% lower inference latency.

Problem

Post-training VLAs via SFT or RL improves task-specific performance but narrows the generalist capacity acquired during large-scale pretraining, and is computationally expensive on billion-parameter backbones. The deeper issue is that VLA failures arise not only from poor action generation but from the absence of action evaluation: the model cannot distinguish high-quality actions it already generates from mediocre or harmful ones in its output distribution. A pass@k diagnostic confirms average success rises from 33% (pass@1) to 92% (pass@32), revealing latent competence that single-shot inference cannot exploit.

Method

SVA operates in three stages on a frozen VLA policy \(\pi_\theta\):

Search. MCTS explores \(\pi_\theta\)'s output distribution in simulation. Each tree edge \((s,a)\) stores a Q-value, visit count, and prior probability. Action selection follows the PUCT rule: $\(a_t = \arg\max_a \left[ Q(s_t, a) + c_\text{puct} P(s_t, a) \frac{\sqrt{n(s_t)}}{1 + n(s_t, a)} \right]\)$ Leaf nodes are expanded by sampling \(N\) candidates from \(\pi_\theta\); each is evaluated via a simulation rollout accumulating discounted return \(G(s_L) = \sum_{i=0}^{D-1} \gamma^i r_{L+i}\), which is backed up as a running-mean Q-estimate along the traversal path.

Value. MCTS-annotated trajectories (both successful and failed, providing contrastive signal) are used to train a lightweight Q-model \(\hat{Q}_\phi\) built on a VLM backbone (Qwen3.5-0.8B) augmented with LoRA adapters and an ensemble of MLP value heads. A special <VALUE> token's hidden state is fed into the ensemble; the model is trained with Smooth-L1 loss: $\(\mathcal{L}(\phi) = \begin{cases} \frac{1}{2}\epsilon^2 & |\epsilon|<1 \\ |\epsilon|-\frac{1}{2} & \text{otherwise} \end{cases}, \quad \epsilon = \frac{1}{B}\sum_{i=1}^B \hat{Q}_\phi^{(i)}(s,a;l) - Q(s,a;l)\)$

Act. At deployment, \(\pi_\theta\) proposes \(N\) candidate actions; the Q-model selects: $\(a_t = \arg\max_{i} \left[ \mu_\phi(s_t, a^{(i)}; l) - \lambda_1 \sigma_\phi(s_t, a^{(i)}; l) + \lambda_2 \log p(a^{(i)} | s_t; l) \right]\)$ where \(\mu_\phi/\sigma_\phi\) are ensemble mean/std and \(\lambda_2 \log p\) is a prior regularizer estimated from sampling frequency. No simulator access is required at deployment.

Key Contributions

  • Identifies the evaluation bottleneck in frozen VLAs: high-quality actions exist in the output distribution but cannot be reliably distinguished at pass@1.
  • Proposes SVA, a three-stage framework (Search → Value → Act) that distills MCTS simulation trajectories into a fast, deployable Q-value model without modifying the VLA backbone.
  • Demonstrates model-agnostic applicability across proprietary (GPT-4o), open-source (Qwen3.5 family), and lightweight (Gemma) VLMs, as well as manipulation VLAs (\(\pi_0\), \(\pi_{0.5}\), OpenVLA).
  • Shows that test-time compute scaling via SVA is more cost-effective than scaling model size (9B+SVA > 27B baseline by 7 points at 27% lower latency).
  • Provides a diagnostic pass@k framework for quantifying the evaluation vs. generation bottleneck split.

Results

  • EmbodiedBench (EB-Habitat / EB-Navigation): SVA yields average gains of +15.4 / +13.2 points across all five backbone models; largest gains on Long Horizon (+23.34 on Qwen3.5-27B EB-Habitat) and Common Sense (+28.33 on Qwen3.5-27B EB-Habitat).
  • SimplerEnv (WidowX): \(\pi_0\)+SVA achieves 50.7% vs. \(\pi_0\)+RoboMonkey's 46.3%; largest per-task gain on Stack Cubes (63.9 vs. 37.5, +26.4 pp).
  • RoboTwin 2.0: \(\pi_{0.5}\)+SVA improves average from 36.0% to 43.5% (+7.5 pp); surpasses RoboMonkey by +5.2 pp; gains span Turn Switch (+20.0), Press Stapler (+16.7), Click Bell (+15.0).
  • Model-size comparison: Qwen3.5-9B+SVA outperforms Qwen3.5-27B baseline by 7 points at 27% lower inference latency.
  • Ablation: Removing the Q-model causes the largest drop (56.11 → 43.33 on EB-Navigation with Qwen3.5-9B); removing MCTS in favor of policy-rollout returns yields a moderate decline; all three components (MCTS, Q-model, multi-candidate) are necessary, with SVA outperforming all ablations by ≥ +5.28 avg.

Limitations

  • MCTS data collection requires simulator access during training; tasks/environments without a simulator cannot use SVA's search stage as described.
  • Gains concentrate in the intermediate-difficulty regime; tasks that exceed base policy capability (e.g., Stack Cubes pass@32 only reaches 50.7%) are not rescued by better evaluation alone.
  • Diminishing returns with increasing \(N\) (marginal gain per additional candidate shrinks from \(\Delta_{1\to2}=0.13\) to \(\Delta_{16\to32}=0.08\)), limiting naive scaling.
  • The Q-model is trained per task distribution; generalization to entirely out-of-distribution tasks (beyond those searched) is not fully characterized.
  • Real-robot evaluation is discussed only in terms of benchmark relevance (Appendix C); direct physical deployment results are not reported in the main text.

Relevance to Vision-Language Models

SVA directly addresses the gap between VLMs' strong generalization and the relative fragility of VLAs, arguing that the core missing piece is a consequence-aware evaluator rather than more backbone parameters—a framing that reframes test-time scaling for embodied VLMs. The approach repurposes a small VLM (Qwen3.5-0.8B) as a Q-model evaluator, connecting MCTS-based credit assignment to the broader trend of using VLMs as reward/value models in agentic settings. It provides a practical blueprint for augmenting any frozen VLM-based policy with long-horizon evaluation without retraining, which is directly relevant to researchers studying test-time compute scaling, process reward models, and best-of-N selection for VLMs. The result that a 9B+SVA system beats a 27B baseline challenges the default assumption that capability scaling requires larger generators rather than better selectors.