Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment

🕒 Published (v1): 2026-07-07 17:27 UTC · Source: Arxiv · Venue: ICML 26 · link

Why this paper was selected

ICML 2026; VLM physical reasoning failure modes directly impact VLA agent generalization

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VAORA (Visual Action Outcome Reasoning Alignment) introduces a dual-reward RL framework that trains VLMs to ground their chain-of-thought reasoning in verifiable visual observations — both before and after an action — for interactive physical reasoning tasks. By suppressing hallucinated CoT and closing the gap between stated reasoning and executed actions, an 8B VLM surpasses the DQN expert it was guided by on unseen PHYRE tasks and transfers zero-shot to a different physics simulator.

Problem

VLMs applied to interactive physical reasoning exhibit two structural failure modes. First, SFT produces fluent but physically inconsistent CoT because reasoning is never checked against observable outcomes. Second, success-driven RL (e.g., GRPO) collapses CoT entirely: the model reverts to shortcut visual-to-action mappings that exploit dataset correlations, with sparse binary rewards in continuous action spaces causing training instability and policy collapse. Both paradigms leave the model's reasoning trace unsupervised with respect to physical reality, yielding brittle cross-task and cross-environment generalization.

Method

VAORA wraps RL training with three symbolic reward signals organized into two categories:

Visual Alignment Reward (\(r_G\)): Before any action, the VLM is prompted to emit structured scene-perception tuples \((\hat\ell^{(i)}, \hat x^{(i)}, \hat y^{(i)}, \hat s^{(i)})\) for each object. The reward combines a soft grid-proximity score between predicted and ground-truth spatial labels with a normalized coordinate/size error: $\(r_G^{(i)} = \frac{1}{2}\left(\text{soft}(\hat\ell^{(i)}, \ell^{(i)}) + 1 - \frac{\|(\hat x^{(i)}, \hat y^{(i)}, \hat s^{(i)}) - (x^{(i)}, y^{(i)}, s^{(i)})\|}{\sqrt{3}}\right)\)$

Visual-Action Alignment Rewards (\(r_C\), \(r_P\)): After executing action \(a\), the post-interaction visual outcome \(v\) is parsed into symbolic ground-truth. \(r_C\) (collision reward) matches predicted collision events — object pairs, action type (COLLIDE/ROTATE), contact point, and motion direction — against simulator-extracted events; \(r_P\) (placement reward) aligns the reasoning's predicted ball placement with its actual location in the first post-interaction frame \(v_0\).

Expert-Guided (EG) dense reward (\(r_\text{DQN}\)): A pre-trained DQN provides smooth success-probability estimates \(\hat p_\text{DQN}(o, a)\) replacing the sparse binary success signal. The visual-action rewards are gated by this estimate to ensure reasoning quality is credited only when the action is physically plausible. Final optimization via GDPO jointly trains: \(r_\text{DQN} \cdot (1 + r_P + r_C)\) and \(r_G\).

Key Contributions

• Diagnostic identification of two distinct VLM failure modes in interactive physical reasoning: hallucinated CoT and reasoning-action misalignment, showing empirically that standard SFT and GRPO each address neither.
• VAORA reward framework with complementary Visual Alignment (\(r_G\)) and Visual-Action Alignment (\(r_C\), \(r_P\)) rewards projecting reasoning and observations into a shared symbolic space.
• Expert-guided dense reward substitution to stabilize RL in continuous action spaces, empirically recovering from the catastrophic degradation observed under raw GRPO.
• Demonstration of cross-task generalization (PHYRE held-out splits), cross-environment zero-shot transfer (PHYRE → Virtual Tool), and transfer to physical reasoning VQA (CRAFT).

Results

PHYRE cross-task generalization (3 held-out splits, Pass@\(k\)): - +EG+VAORA vs. DQN expert: surpasses DQN on nearly all metrics across all 3 testing sets (e.g., Test Set 1 Pass@1: 0.382 vs. 0.326; Pass@5: 0.594 vs. 0.476) - +EG+VAORA vs. Gemini-3.1-Pro: surpasses on Test Sets 1 and 2, competitive on Test Set 3 - Raw GRPO and SFT+GRPO both severely degrade vs. SFT alone, confirming instability under sparse binary rewards

Virtual Tool zero-shot transfer (trained only on PHYRE): - +EG+VAORA achieves best Pass@1 (0.222) among all methods; matches/outperforms Gemini-3.1-Flash across all \(k\); competitive with Gemini-3.1-Pro - DQN expert collapses to near-zero (Pass@1=0.000, Pass@3=0.056), confirming non-VLM agents cannot transfer environments

CRAFT VQA physical reasoning: - +EG+VAORA Overall: 46.06 vs. SFT 44.14 vs. base Qwen3-VL-8B 38.34 - Largest gains in counterfactual (+1.3 pp over SFT) and causal (+1.2 pp) categories — precisely those requiring interactive causal understanding

Reward breakdown ablation: SFT transfers scene-grounding but fails on placement and collision rewards; +EG alone collapses placement entirely on Test Set 1 (0.0); +EG+VAORA substantially recovers all three reward components across all splits.

Limitations

• VAORA relies on a domain-specific simulator to extract ground-truth symbolic states (\(\psi(o)\), \(\psi(v)\)) at training time; real-world deployment without a simulator requires alternative grounding signals.
• The symbolic reward extraction functions \(\phi\) and \(\psi\) are hand-designed for PHYRE's 2D physics; extending to 3D manipulation or unstructured environments requires new symbolic representations.
• The DQN expert used as the density estimator is in-domain (PHYRE-specific), limiting the EG reward's applicability to domains without a strong task-specific agent.
• Table 4 data is partially truncated in the provided text (Testing Set 3 rows incomplete), obscuring full ablation evidence.
• All experiments use an 8B base model; scalability and behavior at larger model sizes are not explored.

Relevance to Vision-Language Models

This work directly targets a core open problem in VLM deployment: chain-of-thought reasoning that is fluent but physically unfaithful and does not drive consistent action. The VAORA reward framework offers a principled alternative to both SFT and task-success RL that grounds CoT in verifiable visual observations, which is broadly applicable to any VLM trained with RL in interactive settings (robotic manipulation, GUI agents, game-playing). The finding that success-driven GRPO degrades reasoning alignment while dense expert-guided rewards recover it has direct implications for the growing literature on VLM post-training via RL. The cross-environment zero-shot transfer results also provide evidence that properly grounded reasoning induces environment-agnostic physical representations rather than benchmark-specific shortcuts.