Skip to content

Environmental Understanding Vision-Language Model for Embodied Agent

🕒 Published (v1): 2026-04-21 09:11 UTC · Source: Arxiv · Venue: CVPR · link

Why this paper was selected

Environmental understanding VLM improves instruction-following embodied agent generalization

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EUEA (Environmental Understanding Embodied Agent) fine-tunes a VLM with four explicit environmental-understanding skills—object perception, task planning, action understanding, and goal recognition—to enable end-to-end instruction-following in embodied environments. A sampling-based recovery step corrects interaction failures without extra training, and a GRPO refinement stage reduces inconsistent skill predictions. On ALFRED, the approach achieves an 8.86% absolute gain over a behavior-cloning baseline, with an additional 3.03% from recovery and GRPO.

Problem

Despite strong perception and reasoning capabilities, VLMs used as embodied agents lack explicit environmental understanding: they fail during object interactions, require privileged environment metadata (object IDs, masks), or depend on complex multi-module pipelines. Existing recovery mechanisms rely on predefined failure types, external environment feedback, or text-only self-feedback that lacks visual grounding.

Method

EUEA defines a reward-free POMDP over instruction-following tasks and internalizes four core skills into a single VLM (InternVL3-8B) via supervised fine-tuning (SFT):

  1. Object Perception: object recognition (OR) lists visible objects; object detection (OD) predicts a bounding box \(b_t = \pi_\theta(I_{OD}, sg_t, a_t, o_t, f_t)\).
  2. Task Planning: subgoal task planning (STP) generates subgoals \(sg_n\) from accumulated memory \(\mathcal{M}\); step-by-step action planning (SAP) generates action-object pairs conditioned on \(k\) past memory steps.
  3. Action Understanding: action success prediction (ASP) predicts whether an action will succeed; future situation captioning (FSC) describes expected post-action state changes (constructed from random-exploration failure data); action grounding (AG) infers which action was executed between two consecutive frames.
  4. Goal Recognition: main goal recognition (GR\(_\text{main}\)) and subgoal recognition (GR\(_\text{sub}\)) predict task/subgoal completion.

All eight sub-skills are fine-tuned jointly in a single stage on 1.24M (ALFRED) and 3.7M (LangR) samples.

Recovery Step: when an interaction fails, the agent samples \(n=10\) alternative action-object pairs via SAP and selects the one with lowest negative log-likelihood score \(s_{t,i} = -\log \pi_\theta(a_{t,i}, o_{t,i} \mid I_{SAP}, m_{t-k:t})\). If all samples repeat the failed action, OD is re-sampled to find an alternative bounding box.

GRPO Stage: a LoRA fine-tune on ~10k "uncertain" instances (those with high reward variance across 8 sampled responses) using task-specific reward functions: Jaccard index for OR, IoU for OD, cosine similarity for STP, action-object correctness for SAP/ASP/AG, and binary accuracy for goal recognition.

Key Contributions

  • Single-VLM framework integrating 4 core environmental-understanding skills (8 sub-skills) without separate module modeling, enabling end-to-end instruction-following.
  • Sampling-based recovery step (no additional training) that handles two failure modes: wrong action-object pair and wrong bounding box.
  • GRPO refinement stage using rule-based rewards to reduce inconsistent skill predictions on compact (~10k) uncertainty-filtered data.
  • Skill datasets: 1.24M samples (ALFRED) and 3.7M samples (LangR); skill evaluation benchmark with 5.3k (ALFRED) and 4.4k (LangR) evaluation instances.
  • Systematic skill-level evaluation of closed-source (GPT-5, GPT-o3, Claude 4.5 Sonnet, Gemini-2.5) and open-source (LLaVA, InternVL, Qwen) VLMs, revealing concrete capability gaps.

Results

  • SFT vs. BC baseline (InternVL3-8B on ALFRED, 429 tasks): 83.45% vs. 74.59% average success rate (+8.86 pp).
  • GRPO stage adds +2.33 pp: 85.78% average success rate.
  • Recovery step (applied at SFT level): 85.78% success, 89.74% goal condition; combined SFT+Recovery+GRPO: 86.48% success, 90.48% goal condition (+3.03 pp combined over SFT alone).
  • Vs. EMMA (prior ALFRED VLM agent): EUEA SFT 83.45% vs. EMMA 67.83%; EUEA GRPO 85.78%.
  • Skill evaluation (ALFRED): Ours achieves 99.40% object grounding, outperforming all closed/open-source baselines on most sub-skills. Gemini-2.5-Pro is the strongest zero-shot model (63.53% grounding, 0.819 planning cosine sim).
  • Step-by-step planning: most open-source VLMs score below 10% without fine-tuning (InternVL3-8B: 4.42%); EUEA reaches 98.53% on ALFRED.

Limitations

  • Navigation phase still relies on PDDL expert actions; only the interaction phase is learned end-to-end.
  • LangR excluded from task evaluation because its trajectories include interactions with non-visible objects, incompatible with the visible-only observation setting.
  • Recovery step incurs \(n=10\) additional forward passes per failed action, increasing inference cost.
  • GRPO dataset (~10k instances) constructed only from the ALFRED validation set; generalization to novel environments is not directly verified.
  • Skill dataset construction requires access to simulation (AI2-THOR, Habitat 2.0) and expert demonstrations; scalability to real-world settings is unclear.

Relevance to Vision-Language Models

This paper directly addresses a practical weakness of VLMs as embodied agents: the gap between language-grounded perception and reliable closed-loop interaction. By decomposing environmental understanding into a structured set of learnable skills and fine-tuning a single VLM to internalize all of them, EUEA provides a blueprint for how VLM fine-tuning can be organized around task-relevant competencies rather than end-task imitation alone. The systematic skill-level benchmarking of GPT-5, Gemini-2.5-Pro, Claude 4.5 Sonnet, and open-source models exposes concrete failure modes (e.g., near-zero step-by-step planning accuracy without fine-tuning) that are directly actionable for VLM researchers. The GRPO stage with task-specific rule-based rewards connects to the broader trend of applying RL-from-rules to stabilize VLM behavior, extending it from reasoning tasks to embodied interaction.