Self-Challenging Language Model Agents¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Self-Challenging Agents (SCA) is a framework that eliminates the need for human-annotated training tasks by having a single LLM agent alternate between two roles: a challenger that autonomously explores tool environments and synthesizes structured tasks, and an executor that trains on those tasks via RL. The key enabler is the Code-as-Task (CaT) formalism, which packages each synthetic task as (instruction, verification function, example solution, failure cases) to ensure feasibility and verifiability. On Llama-3.1-8B-Instruct, SCA achieves a >2× improvement in self-improvement settings with zero human-annotated data.

Problem¶

Training capable multi-turn tool-use agents via RL requires large pools of high-quality tasks with reliable reward signals—but human annotation is expensive, unscalable, and environment-specific. Prior auto-synthesis methods (e.g., PAE) generate tasks only from initial observations without environment interaction, produce tasks lacking executable verifiers, and rely on LLM-as-judge evaluation that is unreliable in partially observable settings.

Method¶

The framework splits a single LLM into two roles operating over the same tool environment:

Challenger role (\(\pi^{task}\)): The agent interacts with the environment through tool calls to probe available state, then outputs a Code-as-Task (CaT) tuple \((\hat{c}, \hat{R}_{\hat{c}}) \sim \pi^{task}(\cdot | o_{0:T}, a_{0:T-1})\). Each CaT contains: (a) a natural-language instruction, (b) an executable Python verification function \(\hat{R}\), (c) an example solution, and (d) three explicit failure cases. Tasks are auto-filtered: only CaTs where the example solution passes \(\hat{R}\) and all failure cases fail \(\hat{R}\) are retained.
Executor role (\(\pi^{exec}\)): Trained on the filtered synthetic task set \(\hat{C}\) via either:
Self-improvement: one-step REINFORCE / Rejection Fine-Tuning on successful rollouts from the same 8B model (equivalent to SFT on \(\hat{R}_{\hat{c}}(s_T, a_T)=1\) trajectories).
Distillation: SFT on all trajectories (including failures) from a stronger 70B teacher sampled over the same synthetic tasks.

Key Contributions¶

Code-as-Task (CaT) formalism: a structured 4-component task representation with executable verification enabling automatic quality filtering without human review.
Environment-exploring challenger: unlike prior proposers that reason only from API docs, the challenger actively queries the environment before generating tasks, producing feasible tasks in partially observable settings.
Algorithm-agnostic RL compatibility: CaT's binary verifier reward works with offline (Rejection FT, DPO) and online (PPO, GRPO) RL algorithms.
Fully unsupervised pipeline: no existing task set \(C\) or human-written verification functions required at any stage.

Results¶

Self-improvement (Llama-3.1-8B on itself): average Pass@1 from 12.0% → 23.5% across 4 environments (+11.5% absolute); outperforms PAE by +10.6% Pass@1.
Distillation (8B student, 70B teacher): +20.2% absolute average Pass@1 vs. zero-shot 8B; beats PAE by up to +5.2% Pass@1 on partially observable environments (Retail).
RL algorithm ablation (Calculation env): Rejection FT: 20.3% → 31.8%; PPO: 43.2%; GRPO reaches similar peaks but collapses to 0% without careful tuning.
PAE failure on Airline (self-improvement): PAE drops to 0% Pass@4; SCA achieves 10.0% Pass@4.
Evaluated on M3 ToolEval (Calculation, Web Browsing) and Tau-Bench (Retail, Airline); all test tasks are out-of-distribution relative to generated training tasks.

Limitations¶

Challenger quality depends on the base model's ability to explore environments reliably; weaker models may generate infeasible tasks despite CaT filtering.
Self-improvement experiments use the same 8B model as both challenger and executor, meaning task diversity is bounded by the base model's knowledge of the environment.
GRPO training is unstable and requires careful hyperparameter tuning to avoid catastrophic collapse.
User simulation in Tau-Bench training uses Llama-3.1-8B instead of GPT-4o (used at evaluation), introducing a distribution mismatch.
Experiments are limited to 4 environments; generalization to qualitatively different tool ecosystems is not demonstrated.

Relevance to Harnesses / Meta-Harnesses¶

SCA is directly relevant as a self-constructing training harness: the challenger role constitutes an automated pipeline that probes environments, generates task specifications, and validates them—functions typically performed by a human-engineered meta-harness. The CaT formalism—combining instruction, verifier, gold solution, and failure cases as a structured unit—maps closely to the design pattern of meta-harness scaffolding that wraps tasks with executable evaluation contracts. The auto-filtering step (run example solution and failure cases through the verifier before accepting a task) is precisely the kind of correctness gating that robust harnesses implement. For researchers building or studying harnesses, SCA demonstrates that the harness specification itself can be generated and validated programmatically by the agent being trained, potentially enabling self-bootstrapping meta-harnesses that require no hand-authored task skeletons.