Skip to content

OpenForgeRL: Train Harness-native Agents in Any Environment

🕒 Published (v1): 2026-07-23 17:38 UTC · Source: Arxiv · link

Why this paper was selected

MSR team; trains agents inside real harnesses (Claude Code, Codex); directly actionable for harness builders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OpenForgeRL is an open-source training framework that enables end-to-end RL training of LLM/VLM agents inside real production inference harnesses (Claude Code, Codex, OpenClaw, etc.) across diverse environments. It uses a lightweight proxy to intercept harness model calls and a Kubernetes orchestrator to run rollouts in remote containers, feeding trajectories into standard RL codebases like veRL without train–deploy mismatch.

Problem

Modern agents depend on sophisticated inference harnesses that manage multi-turn interaction, tool calls, subagents, and long-horizon context — but open RL stacks assume simple, locally-accessible rollouts and cannot natively express stateful multi-process harness inference. Researchers either must re-implement simplified harnesses for training (introducing train–deploy mismatch) or run harness rollouts co-located on training nodes (infeasible at scale). This leaves proprietary systems structurally ahead of what the open research community can train and study.

Method

OpenForgeRL introduces two core components that decouple rollout from training:

  1. Lightweight proxy: Sits between the harness and the inference server (e.g., vLLM). It intercepts every generation request issued by harness containers, routes them to the RL framework's inference engine, and records prompt-response pairs. Upon rollout completion, it reconstructs training trajectories as: $\(\tau = (s^H_0, a_0, r_0), (s^H_1, a_1, r_1), \ldots, (s^H_T, a_T, r_T); \quad r_t = \gamma^{T-t} \cdot r_T\)$ typically with \(\gamma = 1.0\), then passes them to any standard RL algorithm (e.g., GRPO with group-normalized advantages).

  2. Kubernetes orchestrator (built on Orchard): Elastically launches, manages, and tears down rollout container pods on cloud providers (Azure). Each pod runs an isolated sandbox with a pre-installed harness. A wall-clock timeout terminates and discards unresponsive rollouts to prevent training stalls; failed/partial trajectories are discarded rather than used with corrupt credit assignment.

Additionally, a data synthesis pipeline auto-generates SFT and RL tasks for data-scarce domains: agents (1) propose task candidates grounded in real scenarios, (2) prune duplicates/low-quality candidates, (3) build executable Dockerfile-based environments plus verifier scripts, (4) test by rolling out an open LLM, and (5) refine until passing. Training used Qwen3-30B-A3B-Thinking (claw) and an 8B VLM (GUI), fine-tuned first with SFT (distillation from a stronger teacher) then RL via GRPO on 8Ă—B200 GPUs through veRL.

Key Contributions

  • OpenForgeRL framework: plug-and-play proxy + Kubernetes orchestrator that connects any harness Ă— any environment to any RL codebase with no train–deploy mismatch.
  • Broad empirical validation across text-based claw agents (ZeroClaw, OpenClaw, Codex, ReACT) and multimodal GUI agents (browser-use and computer-use), improving over open same-size baselines on nearly all six benchmarks.
  • First open analysis of how harness choice and RL training shape agent behavior at scale: simpler/better-aligned harnesses are easier to learn; RL improves self-verification, tool coverage, and multi-step completion; error recovery remains weak.
  • Automated task/environment synthesis pipeline covering claw, computer-use, and browser-use domains, to be released publicly.

Results

Claw agents (OpenForge-Claw, 30B-A3B MoE, ~3B active): - ClawEval: 31.7 pass3 / 55.9 pass@3 (vs. base Qwen3-30B-A3B-Thinking: 14.3 / 39.8; Qwen3-32B: 6.8 / 31.7) - QwenClawBench: 33.7 pass@1 (vs. Qwen3-30B-A3B-Thinking: 21.8; Qwen3-32B: 14.6) - MCPAtlas: 28.1 pass@1 (vs. Qwen3-30B-A3B-Thinking: 12.4; Qwen3-32B: 22.5) - SFT→SFT+RL improvement on ClawEval: 21.7→31.7 pass3

GUI agents (OpenForge-GUI, 8B VLM): - OSWorld-Verified: 37.7 (vs. UI-TARS-1.5-7B: 27.4; Qwen3-VL-8B: 29.4; MolmoWeb-8B: 34.4 SFT-only baseline from the paper's own SFT) - Online-Mind2Web: 63.0 (vs. Qwen3-VL-8B: 38.7; UI-TARS-1.5-7B: 31.3) - WebVoyager: 72.3 (vs. Qwen3-VL-8B: 49.2; UI-TARS-1.5-7B: 66.4; MolmoWeb-8B: 78.2) - Matches or exceeds models several times larger on most GUI benchmarks - SFT→SFT+RL on OSWorld: 34.4→37.7; Online-Mind2Web: 57.4→63.0; WebVoyager: 61.5→72.3

Limitations

  • Error recovery: Identified explicitly as remaining weak after RL training despite overall reliability gains.
  • Partial-rollout credit assignment: Trajectories that end due to network/harness errors are simply discarded; better credit assignment for partial rollouts is noted as future work.
  • Small training datasets: Only 343–900 RL tasks per domain; scaling behavior with larger datasets is unstudied.
  • Wall-clock timeout heuristic: Timeout thresholds for unresponsive rollouts are set manually and may require tuning per harness.
  • Cloud dependency: The Kubernetes orchestrator is validated on Azure; portability to on-premise or other cloud providers is not systematically evaluated.
  • Text truncation: The paper text was cut before the GUI training details and Section 5 behavioral analysis were fully presented, so some experimental details could not be verified from the provided excerpt.

Relevance to Agentic AI / LLM Agents

OpenForgeRL directly addresses the infrastructure bottleneck blocking open research on harness-based agents — the precise architectural layer (Claude Code, Codex, OpenClaw) responsible for much of recent benchmark progress. By enabling end-to-end RL training in real deployment harnesses rather than simplified proxies, it closes the train–deploy mismatch that gives proprietary systems a structural advantage and makes the behavioral analysis of harness-choice effects possible for the first time. The demonstrated gains (especially the 8B GUI model matching much larger VLMs) suggest that harness-aligned training is a high-leverage lever largely untapped in open agent research. This framework is immediately relevant to anyone training or evaluating LLM agents on tool-use, browser-use, or computer-use tasks.