Skip to content

Hy-Embodied-VLM-1.0: Efficient Physical-World Agents

🕒 Published (v1): 2026-07-14 15:34 UTC · Source: Arxiv · link

Why this paper was selected

Hy-Embodied-VLM-1.0; end-to-end embodied agent with multimodal perception and agentic reasoning; buildable system report

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Hy-Embodied-VLM-1.0 is an efficient embodied foundation model (3B activated parameters via MoE) that extends the VLM paradigm to physical-world agents through a three-level action-centric capability taxonomy. Built on the Hy3-A3B language backbone and Hy-ViT2 vision encoder, it achieves top performance on 19 of 38 embodied benchmarks, outperforming comparably-sized models including Qwen3.6-A3B by 4.4% on average. It delivers near-parity with a prior 32B-activated-parameter model at roughly one-tenth the compute.

Problem

General-purpose VLMs are trained for passive perception and visual QA, not for the action-relevant reasoning physical-world agents require: recognizing affordances and manipulation regions, understanding environment transitions caused by actions, and composing decisions over long horizons with failure recovery. Prior embodied models lack a unified capability framework connecting data construction, training, and evaluation, and strong performance has typically demanded large activated parameter counts (32B+).

Method

The model is built on an MoE architecture (Hy3-A3B language backbone + Hy-ViT2 vision encoder) activating only 3B parameters at inference. Development is guided by an action-centric capability taxonomy with three progressive levels: 1. Action-Relevant State Understanding — physical/semantic perception, spatial reasoning (3D geometry, metric depth, orientation, multi-view correspondence), and robotics-centric understanding (affordances, manipulation regions, functional parts). 2. Action-Transition Reasoning — interaction understanding (participants, semantics, temporal localization), action decision and grounding (next-action selection, trajectory/trace prediction), and physical feasibility with local effects (preconditions, postconditions, immediate state changes). 3. Sequential and Adaptive Reasoning — long-horizon planning/goal decomposition, vision-language navigation (VLN), and reflection/repair/recovery.

Training proceeds in four stages: - Pre-training (inherited from Hy-Embodied-0.5): grounding, depth, segmentation, spatial correspondence, affordance, trajectory. - Mid-training & SFT: adds depth reasoning (from Argoverse2, Matterport3D, nuScenes, ScanNet++, Taskonomy with metric-depth annotations and CoT synthesis), task-conditioned grounding, social interaction data (egocentric videos with structured JSON + QA annotations; AVA v2.2, HICO-DET), object/robot-centric trajectory supervision, and VLN data. - RL via GRPO in two stages: Stage I (reasoning-elicitation) uses broad embodied tasks to surface latent reasoning; Stage II (reward-specialized) applies task-aware embodied rewards targeting specific sub-capabilities across the taxonomy. - Rejection-sampling fine-tuning (RSFT): synthesizes high-quality reasoning traces from Stage I policy, then uses them for reasoning internalization into a non-chain-of-thought deployment model.

Key Contributions

  • Three-level action-centric capability taxonomy linking capability formulation to data construction and benchmark evaluation.
  • Systematic post-training data pipeline covering SFT (depth reasoning, social interaction, trajectory supervision, causal/failure-aware reasoning, VLN) and RL stages across all taxonomy levels.
  • Two-stage RL training (reasoning-elicitation → reward-specialized) with task-aware embodied rewards using GRPO.
  • Rejection-sampling fine-tuning for internalizing reasoning chains into efficient inference.
  • An in-house depth evaluation benchmark (Depth-InHouse, 4,677 MC questions) including counterintuitive cases to distinguish genuine depth reasoning from 2D appearance shortcuts.
  • 38-benchmark evaluation suite spanning all three taxonomy levels.

Results

  • Best performance on 19/38 benchmarks, second on 11 more, out of four compared models.
  • Outperforms Qwen3.6-A3B (also A3B scale) by +4.4% average across all 38 benchmarks.
  • Outperforms NVIDIA Cosmos 3-8B and Embodied-R1.5-8B (both 8B models).
  • +8.4% average improvement over previous-generation Hy-Embodied-0.5 MoT-2B.
  • With 3B activated parameters, achieves performance comparable to the prior Hy-Embodied A32B model (~10× fewer activated parameters).
  • State-of-the-art on R2R-CE benchmark (RGB-only instruction-following navigation setting).
  • Strong zero-shot object-goal navigation on Matterport3D (MP3D).

Limitations

  • Pre-training data is unchanged from Hy-Embodied-0.5; capability gains are entirely from post-training stages, which may bound low-level visual grounding improvements.
  • Evaluation is benchmark-only; no real-robot deployment results are reported.
  • The paper is a technical report from Tencent, not peer-reviewed, and baselines are limited to three models (Qwen3.6-A3B, Cosmos 3-8B, Embodied-R1.5-8B).
  • RL reward design details for task-aware embodied rewards are not fully disclosed in the provided text.
  • Depth-InHouse benchmark is in-house and not publicly released at time of writing, limiting external verification.

Relevance to Vision-Language Models

This work directly extends the VLM paradigm (multimodal perception + language reasoning) into physical-world agency, showing how action-centric data curation and RL with embodied rewards can bridge static visual understanding and dynamic interaction. The three-level taxonomy is a transferable framework for designing embodied VLM training pipelines. The MoE efficiency result (3B activated parameters matching a 32B predecessor) is significant for deployment-constrained VLM research. The two-stage RL approach and rejection-sampling reasoning internalization are broadly applicable training recipes for capability-targeted VLM post-training.