Skip to content

Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR OSU; Agent Data Protocol unifying diverse datasets for fine-tuning LLM agents at scale

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Agent Data Protocol (ADP) is a lightweight Pydantic-schema interlingua that unifies heterogeneous agent training datasets into a single standardized format, then converts them to framework-specific SFT pipelines. It collapses the otherwise-quadratic \(O(D \times A)\) dataset-to-framework conversion effort into linear \(O(D + A)\) via a hub-and-spoke architecture. Fine-tuning on the resulting 1.3M-trajectory corpus yields an average ~20% gain over base models across coding, browsing, tool use, and research benchmarks.

Problem

Despite abundant public agent trajectory data spanning web navigation, software engineering, coding, and tool use, large-scale supervised fine-tuning of LLM agents remains rare. The bottleneck is not data scarcity but fragmentation: each dataset uses its own action space, observation structure, and serialization format. Combining \(D\) datasets across \(A\) agent frameworks requires writing \(D \times A\) bespoke converters, making multi-dataset training prohibitively expensive and non-reproducible at community scale.

Method

ADP is implemented as Pydantic schemas with three core abstractions:

  • Trajectory: a sequence of alternating Action/Observation pairs plus metadata.
  • Actions: three types — APIAction(function, kwargs, description) for tool/API calls; CodeAction(language, content, description) for code execution; MessageAction(content) for NL communication.
  • Observations: TextObservation(source, content) for CLI/environment output; WebObservation(html, axtree, url, viewport_size, image_observation) for browser state.

Conversion follows a three-stage pipeline: 1. Raw → ADP: per-dataset converter maps domain-specific actions/observations to the standardized schema. 2. ADP → SFT: per-agent-framework converter translates ADP trajectories into framework-specific scaffolding (system prompts, action space format, context management) for OpenHands, SWE-Agent, and AgentLab. 3. Quality Assurance: automated validation checks tool-call format correctness, ≥80% thought-paired function calls, and proper conversation termination.

The authors unified 13 existing datasets (AgentInstruct, SWE-Gym, SWE-smith, Mind2Web, Synatra, etc.) into ADP format, subsampled for domain balance, and fine-tuned Qwen2.5-Coder-Instruct (7B/14B/32B) via LLaMA-Factory.

Key Contributions

  • ADP schema: a minimal, expressive Pydantic interlingua covering API/tool use, code execution, web browsing, and NL communication.
  • 13 Raw→ADP dataset converters and 3 ADP→SFT framework converters (OpenHands, SWE-Agent, AgentLab).
  • ADP Dataset V1: 1.3M agent training trajectories, the largest publicly released agent SFT corpus.
  • Demonstrated reduction of conversion effort from \(O(D \times A)\) to \(O(D + A)\).
  • Cross-dataset trajectory analysis revealing action-type distributions and function-thought coverage patterns across 13 datasets.
  • Open-source release of all code, schemas, and datasets.

Results

  • SWE-Bench Verified (SWE-Agent + Qwen-2.5-7B-Coder): 0.4% → 20.2% (+19.8%), outperforming SWE-smith-only training (15.2%) and Claude 3 Opus (15.8%).
  • SWE-Bench Verified (OpenHands + Qwen-2.5-7B-Coder): 2.8% → 20.4% (+17.6%), vs. SWE-Gym-only (10.6%).
  • WebArena (AgentLab + Qwen-2.5-7B-Coder): 4.5% → 21.0% (+16.5%), vs. AgentInstruct-only (17.4%).
  • AgentBench OS (OpenHands + Qwen-2.5-7B-Coder): 3.5% → 27.1% (+23.6%).
  • GAIA (OpenHands + Qwen-2.5-7B-Instruct): 7.3% → 9.1% (+1.8%).
  • Average ~20% gain over corresponding base models across all benchmarks.
  • Cross-task transfer: multi-domain ADP training consistently outperforms single-dataset training on each domain's own benchmark.

Limitations

  • Only 13 source datasets are converted; coverage of multimodal-heavy or proprietary-environment datasets is not demonstrated.
  • The 80% function-thought threshold for quality filtering is a hand-tuned heuristic with no ablation.
  • Subsampling from large datasets (e.g., Orca AgentInstruct at 1M+) to maintain domain balance discards a large fraction of available data; optimal mixture weights are not systematically derived.
  • Evaluation is restricted to Qwen2.5-Coder-Instruct variants; generalization to other base model families is not shown in the presented text.
  • The ADP→SFT conversion is framework-specific, meaning each new agent architecture still requires a new (though single) conversion script.
  • Paper text is truncated; Tables 4–5 (14B/32B results) and stated limitations section are not available for full assessment.

Relevance to Harnesses / Meta-Harnesses

ADP is structurally a data-layer meta-harness: it introduces an intermediate representation that decouples upstream data producers from downstream training pipelines, exactly the hub-and-spoke design principle underlying meta-harnesses in execution systems. The ADP→SFT stage functions as a harness adapter layer — converting normalized trajectories into framework-specific scaffolding (system prompts, action serialization, context windowing) for each agent harness (OpenHands, SWE-Agent, AgentLab) without modifying the shared upstream data. The linear \(O(D+A)\) amortization argument is directly analogous to harness design rationale: write one adapter per component rather than one glue-script per pair. For researchers building or extending training harnesses, ADP provides a concrete existence proof that standardizing the intermediate representation at the trajectory level is sufficient to unlock cross-domain transfer without per-dataset engineering.