Libra-VLA: Achieving Learning Equilibrium via Asynchronous Coarse-to-Fine Dual-System¶

🕒 Published (v1): 2026-04-27 19:02 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Asynchronous coarse-to-fine dual system balances VLA learning equilibrium; ACL 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Libra-VLA introduces a coarse-to-fine dual-system VLA architecture that decomposes robotic action generation into a discrete Semantic Planner (macro-directional intent) and a continuous diffusion-based Action Refiner (fine-grained alignment), connected asynchronously via an intent buffer. The key empirical finding is that model performance follows an inverted-U curve with respect to action discretization granularity, peaking when learning difficulty is balanced between the two sub-systems. On LIBERO, Libra-VLA achieves 97.2% average success rate, surpassing all prior baselines including $\pi_0$ and GR00T-N1.

Problem¶

Prevailing VLA models map visual-linguistic features directly to high-frequency motor commands in a flat, non-hierarchical fashion. This enforced uniformity creates a severe semantic-actuation gap: the model must simultaneously handle abstract semantic reasoning and precise low-level control in a single pass. Existing hierarchical approaches decompose along the temporal axis (sub-goals, keypoints, sub-instructions) but still require a cross-modal translation to continuous actions at each step, leaving single-step representational complexity unresolved.

Method¶

Libra-VLA factorizes the joint action distribution as: $$P(a_t | o_t, L) \approx \underbrace{P(a_t^f | a_t^c, o_t)}_{\text{Action Refiner}} \cdot \underbrace{P(a_t^c | o_t, L)}_{\text{Semantic Planner}}$$

System 2 — Semantic Planner (low frequency): A VLM backbone (InternVL2.5-2B) augmented with a Parallel Coarse-Action Head. Continuous actions in $[-1,1]^D$ are uniformly quantized into $N \ll 256$ bins; the paper uses small $N$ to deliberately capture macro-directional intent rather than precise kinematics. Learnable query tokens $Q_{act} \in \mathbb{R}^{K \times D}$ are concatenated with VLM features and passed through a bidirectional transformer; outputs are projected to a softmax over $N$ bins per dimension. Training uses cross-entropy loss $\mathcal{L}_{plan} = \mathcal{L}_{CE}(P(a_t^c), y_t^{gt})$.

System 1 — Action Refiner (high frequency): A diffusion transformer conditioned on the coarse intent embedding $e_{intent}$ retrieved from a learnable codebook $E \in \mathbb{R}^{N \times D}$, plus geometric features $F_t^{geo}$ from an independent SigLIP visual encoder. Predicts denoising noise $\epsilon_\theta(x_k, F_t^{geo}, e_{intent})$; trained with MSE loss. A dynamic curriculum strategy switches $e_{intent}$ from ground-truth tokens (early training, when System 2 success rate is below threshold $\tau$) to sampled predicted tokens, building error-correction robustness.

Asynchronous Execution: System 2 predicts a macro-horizon $L_{macro} = M \times H_{chunk}$ coarse tokens in one pass, stores them in a FIFO intent buffer, then stays dormant for $M{-}1$ subsequent System 1 cycles. System 1 pops $H_{chunk}$ tokens per step. Experiments use $M=2$, $H_{chunk}=5$.

Training objective: $\mathcal{L}_{total} = \lambda_{diff}\mathcal{L}_{diff} + \lambda_{plan}\mathcal{L}_{plan}$

Key Contributions¶

Hybrid Action Space formulation: decomposes robotic actions into a discrete macro-directional subspace and a continuous micro-pose subspace, providing an intra-modal progressive bridge across the semantic-actuation gap
Identification of the learning complexity equipartition principle: performance peaks at an intermediate bin count $N$ (verified empirically via ablation over $N \in \{2, 10, 50, 100\}$), following an inverted-U curve
Decoupled asynchronous dual-system architecture with a predictive intent buffer, enabling the Semantic Planner to provide temporally synchronized future context rather than static latent embeddings
Independent visual encoder in the Action Refiner, avoiding feature-squeezing bottlenecks present in architectures that force shared weights to encode both semantic and geometric features

Results¶

LIBERO benchmark (500 rollouts, 50 per task, no large-scale robot data pretraining): - Libra-VLA: 97.2% average (Spatial 98.6%, Object 99.4%, Goal 98.0%, Long 92.8%) - Best prior: GE-Act 96.5%, $\pi_0.5$ 96.9%, F1 95.7% - Outperforms all discrete (CoT-VLA 81.1%, DD-VLA 96.3%) and continuous ($\pi_0$ 94.1%, GR00T-N1 93.9%) baselines

LIBERO-Plus benchmark (7 perturbation dimensions): - Zero-shot transfer: 79.5% average vs. next-best OpenVLA-OFT 69.6% - Supervised fine-tuning: 82.3% average vs. OpenVLA-OFT+ 79.6%, $\pi_0^*$ 67.4%

Ablation (LIBERO): - Libra-Base (monolithic): 88.3% avg - Libra-VE (add visual encoder only, no coarse-to-fine): 87.0% avg — performance drops, especially Goal (69.2%), attributed to visual shortcut learning - Libra-Refinement (coarse-to-fine without independent VE): 95.1% avg - Full Libra-VLA: 97.2% avg

Limitations¶

Evaluation conducted exclusively on simulation benchmarks (LIBERO, LIBERO-Plus) and unspecified real-world tasks; large-scale real-robot benchmarking is absent
The optimal bin count $N$ is task/dataset dependent; the paper acknowledges the inverted-U curve but does not provide an analytical criterion for selecting $N$ a priori
No large-scale robot-data pretraining is used, which may limit comparison fairness with models like $\pi_0.5$ that leverage such pretraining
The paper text is truncated before full ablation results for training strategy and asynchronous execution ($M$ sweep) are presented

Relevance to Vision-Language Models¶

Libra-VLA directly addresses the bottleneck of grounding VLM representations (trained on discrete text tokens) to continuous motor control, a core challenge in VLA research that inherits VLM architectures. The architecture demonstrates that VLMs' native discrete reasoning can be systematically leveraged for coarse action planning without forcing the backbone to also handle metric precision, providing a principled design for VLM-to-robot grounding. The learning complexity equipartition principle offers a quantitative insight into how the semantic capacity of a VLM couples with downstream control tasks. For VLM researchers, the asynchronous dual-system design also exemplifies how VLM inference frequency can be decoupled from robot control frequency, a practical consideration for deploying large VLMs in real-time embodied settings.