Skip to content

Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning

🕒 Published (v1): 2025-12-10 14:52 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CrossHA is a unified agentic model trained via a three-stage pipeline (SFT → single-turn GRPO → multi-turn GRPO) to autonomously select among heterogeneous action spaces (high-level APIs, GUI events, low-level motor primitives) at each step of a trajectory. Rather than fixing an agent to one action space, CrossHA learns when to use which interface as a policy-level decision. It achieves state-of-the-art on 800+ Minecraft tasks despite training on only 30.

Problem

Existing agents are confined to a single, statically defined action space (API calls, GUI events, robotic primitives, etc.), which creates two failure modes: (1) brittle dependencies on specific translation layers (e.g., a scraping API blocked by CAPTCHA), and (2) inability to adapt action granularity within a single task where optimal interface varies step-by-step. Prior multi-space agents that mix action types still rely on hand-engineered switching rules rather than learned policies.

Method

CrossHA frames action-space selection as a learnable component within an MDP over a composite action space A = ∪A_x. Training proceeds in three stages:

  1. Mixed-Space SFT (M_mix): Fine-tune Qwen2-VL-7B on a balanced dataset of trajectories from all action subspaces; model learns syntax/semantics of each space but selects stochastically.

  2. Single-Turn RL (STRL, M_strl): Warm-up via rejection sampling to generate valid multi-space candidates, then optimize with GRPO using an action-space-agnostic reward: credit is given whenever the parsed raw action matches ground truth regardless of which surface space produced it. This teaches strategic rather than stochastic space selection on single-step problems.

  3. Multi-Turn RL (MTRL, M_mtrl): Self-train to initialize from M_strl's preferences (relabeling dataset D_strl), then run online GRPO with a binary episodic success reward penalized by total token length (λ·l_θ(τ)). The token-length penalty causes the agent to prefer concise high-level actions when they suffice, and switch to verbose low-level actions only when necessary.

Key Contributions

  • CrossHA: a single policy model that masters heterogeneous action spaces and autonomously selects the appropriate interface per trajectory step without human-specified rules.
  • Multi-Turn GRPO training pipeline enabling trajectory-level action-space optimization, with a token-length penalty that incentivizes efficiency as emergent behavior.
  • Empirical demonstration that treating action-space selection as a learnable policy (via RL) outperforms hand-engineered switching in both success rate and generalization.
  • Strong OOD generalization: trained on 30 tasks, evaluated on 800+; smaller ID→OOD performance gap than single-space RL baselines.

Results

  • CrossHA overall ASR: 54.6% across all 800+ OpenHA Minecraft tasks; second-best is Game-TARS at 45.1%.
  • Mine Blocks FT: 94.7% (best); ASR: 66.6%.
  • Craft Items FT: 83.3% (best); ASR: 78.8% on OOD tasks vs. 69.8% for next-best RawHA-RL.
  • Kill Entities ASR: 52.1%, best among all agents.
  • OOD All Tasks ASR: 49.1% vs. 42.4% (RawHA-RL), 39.4% (GroundingHA-RL), 39.1% (MotionHA-RL).
  • Ablation: removing STRL degrades OOD Craft Items from 78.8% to 58.0% and All Tasks from 49.1% to 39.7%.
  • Single-space baselines (e.g., RawHA-RL 96.2% on Craft Items ID) drop sharply OOD (69.8%), while CrossHA maintains smaller generalization gaps.

Limitations

  • Evaluation is limited to Minecraft; generalization to real-world robotics or digital computer-use tasks is untested.
  • MTRL is computationally expensive: 80+ iterations, each requiring 6,400+ environment interactions.
  • The composite action space requires separate controllers/interfaces per subspace (e.g., grounding pipeline, motion-generation module), which must be pre-built.
  • Token-length penalty as efficiency proxy may not align with actual execution latency or resource cost.
  • Only three action-space types are explored in Minecraft; the framework's scaling to richer heterogeneous spaces (e.g., MCP + GUI + robotics) is unvalidated.

Relevance to Agentic AI / LLM Agents

CrossHA directly addresses the rigid action-space constraint that limits deployed LLM/VLM agents, showing that action-space selection itself can and should be a learned policy rather than an engineered rule. The Multi-Turn GRPO formulation is directly applicable beyond games—any agentic system facing a choice between tool-call APIs, GUI manipulation, and shell commands (e.g., computer-use agents) faces the identical structural problem. The emergent efficiency behavior (preferring concise interfaces when sufficient) is a notable result for long-horizon agent design, where token/step costs compound. This work positions RL post-training as the mechanism to unify previously siloed agent paradigms.