Skip to content

Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment

🕒 Published (v1): 2026-07-07 17:27 UTC · Source: Arxiv · Venue: ICML 26 · link

Why this paper was selected

ICML 2026; VLM physical-reasoning alignment with grounded CoT — strong venue, directly relevant to VLM agent scaffolding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VAORA (Visual Action Outcome Reasoning Alignment) is a reward-design framework for training VLMs on interactive physical reasoning. It introduces two complementary reward signals that suppress hallucinated chain-of-thought and close the gap between a model's stated reasoning and its executed actions. Training on PHYRE with VAORA enables an 8B model to surpass the in-domain DQN expert on unseen tasks and achieve zero-shot transfer to a different physics engine.

Problem

VLMs applied to interactive physical reasoning suffer from two structural failure modes: (1) hallucinated CoT — reasoning traces that are linguistically fluent but contradict physical reality, and (2) reasoning-action misalignment — the model produces physically plausible reasoning yet executes an inconsistent action, effectively bypassing CoT via a visual shortcut. SFT teaches imitation of linguistic form without grounding to observable outcomes; success-driven RL (e.g., GRPO) encourages shortcut visual-to-action mappings and causes training instability under sparse binary rewards in continuous action spaces.

Method

VAORA projects both the model's reasoning trace \(r\) and visual observations into a shared symbolic space and measures their consistency: $\(r_{\text{VAORA}} = R(\phi(r),\, \psi(o),\, \psi(v))\)$ where \(\phi(r)\) parses structured symbolic propositions from the CoT, \(\psi(o)\) extracts ground-truth states from the initial scene, and \(\psi(v)\) extracts states from the post-interaction outcome.

Three reward signals are defined:

  1. Grounding reward \(r_G\) (Visual Alignment): for each object \(i\), combines a soft spatial-label score and coordinate/size deviation between the model's predicted tuple \((\hat{\ell}^{(i)}, \hat{x}^{(i)}, \hat{y}^{(i)}, \hat{s}^{(i)})\) and the ground-truth from the initial scene — suppresses hallucinated scene perception independent of any action.

  2. Collision reward \(r_C\) (Visual-Action Alignment): aligns predicted collision events (object identities, action type, contact point, direction) with post-interaction simulator output; contact-point accuracy is \(1 - \frac{\sqrt{(\hat{c}_x - c_x)^2 + (\hat{c}_y - c_y)^2}}{\sqrt{2}}\).

  3. Placement reward \(r_P\) (Visual-Action Alignment): aligns the model's predicted ball-placement spatial label (relative to reference objects) with the actual placement in the first post-interaction frame using the same soft grid score.

To handle sparse rewards in continuous action space, the binary task-success signal is replaced by a smooth, dense Expert-Guided (EG) reward: $\(r_{\text{DQN}}(a) = \hat{p}_{\text{DQN}}(o, a)\)$ from a pre-trained DQN. The visual-action alignment rewards are gated by \(r_{\text{DQN}}\) to reward reasoning quality only when the action is physically plausible. The full composite reward optimized via GDPO is: $\(r_{\text{DQN}} \cdot (1 + r_P + r_C) \quad \text{and} \quad r_G\)$

Key Contributions

  • Identifies the two root failure modes (hallucinated CoT, reasoning-action misalignment) shared by both SFT and success-driven RL paradigms.
  • VAORA dual-reward framework: Visual Alignment Reward (\(r_G\)) and gated Visual-Action Alignment Reward (\(r_C + r_P\)) providing explicit supervision between CoT and observable physical consequences.
  • Expert-Guided (EG) dense reward from a pre-trained DQN stabilizes RL training in continuous action spaces.
  • Demonstrates that an 8B open VLM trained on PHYRE achieves zero-shot cross-environment transfer to Virtual Tool, matching or exceeding frontier closed-source models.

Results

Cross-task generalization (PHYRE, 3 held-out testing sets): - +EG+VAORA achieves Pass@1 / Pass@3 / Pass@5 of 0.382 / 0.524 / 0.594 (Set 1), 0.198 / 0.390 / 0.472 (Set 2), 0.300 / 0.456 / 0.512 (Set 3), surpassing all open-source VLM baselines and most closed-source APIs (Claude Sonnet-4.6, GPT-5.4, Gemini-3.1-Flash). - Surpasses the DQN expert (Pass@1=0.326/0.260/0.338 across sets) on nearly all metrics. - Outperforms Gemini-3.1-Pro on Sets 1 and 2; competitive on Set 3. - GRPO alone degrades significantly vs. SFT baseline (e.g., Set 1 Pass@1: SFT=0.176, GRPO=0.006), confirming training collapse under binary rewards.

Cross-environment generalization (Virtual Tool, zero-shot): - +EG+VAORA: Pass@1 / Pass@3 / Pass@5 = 0.167 / 0.222 / 0.333 — best Pass@1 among all models; DQN collapses to 0.000/0.056/0.056. - Competitive with Gemini-3.1-Pro (0.111/0.278/0.389) despite 8B scale and no Virtual Tool exposure.

Broader physical understanding (CRAFT VQA): - +EG+VAORA overall accuracy 46.06 vs. SFT baseline 44.14, base model 38.34. - Largest gains in counterfactual (+1.3pp vs. SFT) and causal categories (+1.2pp), indicating genuine causal reasoning transfer.

Reward breakdown analysis (Table 4): - SFT transfers scene grounding (0.710) but fails on placement (0.348) and collision (0.255) on Set 1. - +EG collapses placement (0.0 on Set 1) and grounding (0.424), indicating shortcut regression. - +EG+VAORA recovers grounding (0.753) and substantially improves placement (0.586) and collision (0.511).

Limitations

  • Requires a pre-trained in-domain expert (DQN) to supply the dense reward signal; this expert must be re-trained per physics environment for the EG component.
  • Domain-specific symbolic transformations \(\phi\) (CoT parser) and \(\psi\) (scene/outcome extractor) must be re-engineered for new environments; the paper only instantiates them for 2D ball-placement simulators.
  • Evaluation restricted to 2D physics simulations (PHYRE, Virtual Tool); generalization to richer 3D or real-world settings is unverified.
  • The symbolic space uses a coarse \(3 \times 3\) spatial grid, which may limit precision for fine-grained physical manipulation.
  • Results are based on Qwen3-VL-8B as the sole backbone; sensitivity to model scale or architecture is not studied.

Relevance to Agentic AI / LLM Agents

VAORA directly targets a core weakness of LLM-based agents: the decoupling of verbalized reasoning from actual action consequences. By introducing post-interaction visual outcomes as an explicit RL supervision signal, it advances a principled methodology for grounding agent CoT in environmental feedback — a prerequisite for reliable agentic behavior. The zero-shot cross-environment transfer result is notable for agent generalization research, demonstrating that reward alignment (not environment-specific fine-tuning) can be the key driver of transferable physical understanding. The finding that standard success-driven RL (GRPO) causes policy collapse while visual alignment rewards stabilize training has direct implications for how RL is applied to train VLM-based tool-using or embodied agents.