Skip to content

Let It Flow: Agentic Crafting on Rock and Roll, Building the ROME Model within an Open Agentic Learning Ecosystem

🕒 Published (v1): 2025-12-31 14:03 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces the Agentic Learning Ecosystem (ALE), a full-stack open-source infrastructure for training and deploying agentic LLMs, consisting of three tightly integrated components: ROLL (RL training framework), ROCK (sandboxed environment execution engine), and iFlow CLI (agent framework/context manager). Built on ALE, the authors train ROME, a Qwen3-MoE-based open-source agent on 1M+ trajectories. ROME achieves 57.40% on SWE-bench Verified and 24.72% on Terminal-Bench 2.0, rivaling models with >100B parameters.

Problem

The open-source community lacks a principled, end-to-end agentic ecosystem that closes the loop across data generation, environment execution, and policy optimization. Prior work relied on either SFT from limited human demos or ad-hoc RL recipes that struggle with long-horizon, sparse-reward tasks. No existing open infrastructure co-designs training framework, sandboxed environments, and deployment context management, leading to train/deploy inconsistency and poor production adoption.

Method

ALE integrates three systems:

ROLL (RL training framework): Decomposes agentic RL into worker roles (LLM inference, environment interaction, reward computation, parameter updates) that scale independently. Implements fine-grained rollout by pipelining LLM generation, environment interaction, and reward computation at sample level. Introduces asynchronous training with a sample buffer and "asynchronous ratio" (max allowed policy version gap per sample) to overlap rollout and training. Uses time-division train-rollout GPU multiplexing—dynamically reallocating GPUs between stages based on current bottleneck—to reduce idle bubbles.

ROCK (environment execution engine): Client-server sandboxed environment manager. Exposes a GEM-standard API (make/reset/step/close) that is framework-agnostic. An Admin control plane handles provisioning and scheduling; Worker nodes manage local sandboxes; Rocklet proxies mediate agent↔sandbox communication and enforce egress policies. Scales to 10,000+ concurrent environments. Includes an Agent Native Mode: a ModelProxyService intercepts LLM requests inside the sandbox (already carrying iFlow CLI-assembled context) and forwards them to the appropriate inference backend, cleanly decoupling context management from the training engine.

iFlow CLI (agent framework): Single-agent orchestrator-worker loop following Anthropic's agentic-system recommendations. Provides five context engineering techniques: persistent memory (todo file), context isolation (sub-agents with bounded context), context retrieval (semantic search, DeepWiki), context compression (lossy/lossless), and context enhancement (explicit signal injection). Open configuration via system prompt, workflow/spec definitions, and MCP tool integration.

ROME training pipeline: Three-stage curriculum—(1) Agentic Continual Pre-training (CPT) on code-centric + general reasoning data; (2) SFT to anchor the policy in reliable regions; (3) Interaction-Perceptive Agentic Policy Optimization (IPA), a novel RL algorithm that models multi-turn tasks as Chunked MDPs, assigning credit over semantic interaction chunks rather than individual tokens. IPA uses chunk-level initialized resampling (CIR) to restart rollouts mid-trajectory from a promising chunk, improving exploration efficiency for long-horizon tasks.

Data composition uses a two-tier curriculum: Basic Data (code-centric corpora + general reasoning) and Agentic Data (executable instances + multi-turn trajectories with runtime verification). Safety, security, and validity filtering is enforced via ROCK's sandboxed execution.

Key Contributions

  • ALE: First open-source, production-grade, end-to-end agentic ecosystem unifying data synthesis, RL training, environment management, and deployment.
  • ROLL: Scalable agentic RL framework with fine-grained rollout pipelining, asynchronous training, and dynamic GPU multiplexing.
  • ROCK: Scalable, secure sandboxed environment manager with a GEM-standard API and Agent Native Mode that eliminates train/deploy context inconsistency.
  • iFlow CLI: Context-engineering-first agent framework with five built-in context management techniques and open configuration for domain specialization.
  • IPA algorithm: Chunk-level credit assignment over semantic interaction units, improving long-horizon RL training stability.
  • ROME: Open-source agentic model (Qwen3-MoE base) trained on 1M+ trajectories, achieving 57.40% SWE-bench Verified and 24.72% Terminal-Bench 2.0.
  • Terminal Bench Pro: New benchmark with improved scale, domain coverage, and contamination control over Terminal-Bench 2.0.

Results

  • SWE-bench Verified: 57.40% — outperforms similarly sized open-source models; rivals >100B parameter models.
  • Terminal-Bench 2.0: 24.72%.
  • Terminal Bench Pro: ROME achieves competitive performance among open-source models of similar scale (specific number not provided in excerpt).
  • ROME is deployed in production via iFlow CLI, validating practical effectiveness.
  • ROLL-Flash (prior work) shows asynchronous training balances accuracy and throughput (referenced, not detailed here).
  • Rollout stage historically accounts for ~70% of RL post-training overhead; environment interaction alone >15%; ALE's multiplexing and async design directly address these bottlenecks.

Limitations

  • ROME's base model is Qwen3-MoE; generalizability of ALE to other base architectures is not demonstrated.
  • Terminal Bench Pro results lack specific numbers in the provided text—only "competitive" characterization given.
  • The IPA algorithm's chunk-level credit assignment relies on semantic segmentation of trajectories; the robustness of this segmentation across diverse task types is not analyzed.
  • Data composition pipeline is partially proprietary in implementation detail (e.g., LLM-based judge, expert inspection stages are described but not reproducibly specified).
  • Safety-aligned data and red-team data composition are mentioned but not deeply evaluated in the excerpt.
  • Agent Native Mode requires iFlow CLI to be the context manager; supporting other agent frameworks (SWE-Agent, OpenHands) as primary context managers adds integration complexity.

Relevance to Harnesses / Meta-Harnesses

ALE is a canonical example of a meta-harness: it is not itself a task-solving agent but rather the infrastructure layer that orchestrates data synthesis pipelines, sandboxed execution environments, RL training loops, and deployment scaffolding into a single closed-loop system. The ROCK environment engine with its GEM-standard API is directly analogous to the "environment abstraction layer" pattern seen in harness design—providing a uniform interface across heterogeneous task environments that any RL framework can plug into. The Agent Native Mode (ModelProxyService) solves the train/deploy consistency problem that harnesses face when the inference-time scaffolding (context assembly, prompt structure) must exactly match training-time scaffolding—a critical invariant for harness-trained models. IPA's chunk-level credit assignment is a harness-level design choice: the training harness segments trajectories into semantic units before computing gradients, meaning the harness architecture directly shapes the optimization objective.