Skip to content

Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots

🕒 Published (v1): 2026-07-02 17:58 UTC · Source: Arxiv · link

Why this paper was selected

Portable VLA/WAM inference runtime; directly addresses real-world embodied AI deployment gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Embodied.cpp is a portable C++ inference runtime designed specifically for embodied AI models (VLA models and world-action models) deployed on heterogeneous edge devices. It introduces a five-layer architecture that captures a shared execution path across diverse embodied model families while supporting modular multi-rate execution, latency-first batch-1 inference, and extensible embodied I/O. Two VLA deployments achieve 100% and 91% task success rates with the full C++ runtime stack.

Problem

Existing inference runtimes (llama.cpp, ONNX Runtime, SGLang, vLLM) are designed for request-response serving with throughput-oriented optimization and uniform token interfaces. They fail the "runtime contract" of embodied deployment: (1) components must execute at different rates within a single closed-loop control step; (2) optimization target is batch-1 latency/jitter stability rather than throughput; (3) I/O is heterogeneous—proprioception, force/tactile signals, action chunks, predicted futures—not fixed token streams. The closest prior work, vla.cpp, handles VLA models but not world-action models (WAMs), and lacks modular multi-rate scheduling and direct simulator/robot deployment adapters.

Method

Embodied.cpp is implemented in C++ and organized as a five-layer architecture: 1. Input adapters — typed interfaces absorbing sensor streams (cameras, IMU, force/tactile) and offline datasets 2. Sequence builders — construct per-model input sequences from heterogeneous modalities 3. Backbone execution — shared transformer-style execution path, backend-agnostic, targeting CPU/GPU/NPU via one abstraction 4. Head plugins — pluggable action heads, diffusion/flow heads, world-prediction branches, and predictive modules that diverge across VLA/WAM families 5. Deployment adapters — output bridges to simulators (ManiSkill, LIBERO, Isaac Sim) and real-robot software stacks

Supporting the five layers are three runtime subsystems: (a) a modular multi-rate execution scheduler that decouples components with different refresh rates (e.g., perception encoder vs. high-rate action head); (b) latency-first fused execution with graph replay, buffer reuse, operator fusion, and careful host-device data movement; (c) an embodied AI kernel warehouse of reusable and model-specific operators.

WAM inference additionally uses GGUF Q4_K quantization to compress transformer block weights. The architectural taxonomy (Table 1) categorizes eight embodied model subtypes (AR-Token VLA, VLM-Backboned VLA, Hierarchical VLA, Asynchronous VLA; Predict-then-Act WAM, Unified AR-Modeling WAM, Shared-Backbone WAM, Latent-space WAM) and identifies that the main structural divergences are confined to pluggable heads and predictive modules, justifying the shared-core design.

Key Contributions

  • Architectural taxonomy of embodied models (8 subtypes across VLA and WAM families) revealing a shared backbone execution path with divergences isolated to pluggable heads
  • Design and implementation of Embodied.cpp: a five-layer portable C++ runtime addressing multi-rate execution, batch-1 latency-first control, and extensible embodied I/O on heterogeneous edge hardware (Jetson, RK-based, x86, workstation)
  • Closed-loop VLA evaluation of HY-VLA (Hunyuan-VL backbone) and pi0.5 (PaliGemma backbone) through the full C++ deployment path
  • WAM microbenchmark on LingBot-VA demonstrating Q4_K quantization reduces per-block memory from 312.2 MiB to 88.1 MiB with minimal output drift

Results

  • HY-VLA (Hunyuan-VL backbone, 20-token action chunk, three-view inputs): 100.0% task success rate on RoboTwin place_empty_cup; step latency 735.9 ms, inference latency 1340.3 ms, VRAM 6850 MiB
  • pi0.5 (PaliGemma backbone, 50-token action chunk): 91.0% task success rate [CI: 86–94%]; step latency 56.85 ms, inference latency 266.6 ms, VRAM 6546 MiB
  • LingBot-VA single Transformer block (WAM microbenchmark, 100 random input samples): Q4_K reduces resident weight memory 312.2 MiB → 88.1 MiB (71.8% reduction); MAE \(< 3.3 \times 10^{-2}\), cosine similarity \(> 9.997 \times 10^{-1}\) vs. BF16 Python baseline; latency 3.171 ms vs. 3.236 ms (Python BF16)
  • No full closed-loop WAM results are reported; the LingBot-VA result is a single-block microbenchmark only

Limitations

  • Full closed-loop WAM evaluation is absent; the LingBot-VA result covers only one Transformer block due to model instability on the constrained local edge setup
  • HY-VLA inference latency (1340 ms) reflects a large backbone and multi-view inputs; whether this meets real-time control requirements for all robot platforms is not analyzed
  • No ablation separating the contribution of each architectural layer or runtime subsystem to latency/accuracy
  • Heterogeneous hardware experiments (Jetson, RK boards, NPU backends) are referenced in the design but not benchmarked in the reported evaluation
  • The architectural taxonomy excludes offline or non-online embodied model families; scope is limited to closed-loop inference settings

Relevance to Vision-Language Models

Embodied.cpp directly addresses the deployment gap for VLM-backboned VLA models—specifically models like pi0.5 (PaliGemma) and HY-VLA (Hunyuan-VL) that use large pretrained VLMs as perception-action backbones—by providing a C++ runtime that preserves task accuracy while enabling edge deployment without Python stacks. For researchers tracking VLMs, this work highlights a practical systems bottleneck that limits the real-world impact of state-of-the-art VLA models: the mismatch between VLM serving infrastructure (throughput-first, uniform token I/O) and embodied control requirements (batch-1 latency, heterogeneous I/O, multi-rate scheduling). The five-layer abstraction provides a template for how VLM backbones can be decoupled from model-specific action heads, which has implications for modular VLM reuse across embodied architectures.