Skip to content

Co-Evolving Agents: Learning from Failures as Hard Negatives

🕒 Published (v1): 2025-11-27 09:30 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A co-evolving dual-agent framework in which an auxiliary failure agent is trained via DPO exclusively on failure trajectories to generate high-reward, near-success hard negatives, which are then injected into a target agent's preference optimization loop. The mutual co-evolution sharpens the target agent's preference landscape and improves generalization without external annotation. On three multi-turn benchmarks the framework outperforms the strongest DPO baseline (ETO) by an average of +5.8 reward points on Llama-2-7B.

Problem

Self-improving agents that use Exploration-Based Trajectory Optimization (ETO) pair raw predicted failures with expert trajectories for DPO. Because these failures are of uniformly low quality (far from success), the dispreferred side offers weak contrast, causing the model to collapse toward memorizing the scarce expert demonstrations rather than learning a fine-grained preference landscape. The result is overfitting and poor generalization, especially in tasks where the pretrained LLM has limited prior knowledge.

Method

Both agents are seeded from the same SFT-initialized base policy \(\pi_\text{base}\).

Failure agent (\(\pi_{\theta_f}\)): trained via DPO on pairs drawn from \(D_\text{fail} = \{(u, e^+, e^-) \mid e^+, e^- \in F_\text{tgt} \times F_\text{fail}\}\), where \(e^+\) is a higher-reward failure and \(e^-\) a lower-reward one. The objective is: $\(\mathcal{L}_\text{DPO}(\theta_f) = -\mathbb{E}\left[\log \sigma\!\left(\beta \log \frac{\pi_{\theta_f}(e^+|u)}{\pi_\text{ref}(e^+|u)} - \beta \log \frac{\pi_{\theta_f}(e^-|u)}{\pi_\text{ref}(e^-|u)}\right)\right]\)$ This drives the failure agent to model fine-grained distinctions within the failure landscape and emit trajectories with reward near but below 1.

Target agent (\(\pi_{\theta_t}\)): optimized with a weighted DPO + auxiliary SFT loss: $\(\mathcal{L}_\text{target}(\theta_t) = \lambda_\text{DPO}\,\mathcal{L}_\text{DPO}(\theta_t) + \lambda_\text{SFT}\,\mathbb{E}_{(u,e^+)\sim D_\text{tgt}}\!\left[-\log\pi_{\theta_t}(e^+|u)\right]\)$ with \(\lambda_\text{DPO}=1.0\), \(\lambda_\text{SFT}=0.1\). Its preference dataset \(D_\text{tgt}\) combines expert-vs-target, expert-vs-failure-agent, and failure-vs-failure comparisons. Training alternates between the two agents for 3–5 co-evolution iterations.

Key Contributions

  • Failure agent: an auxiliary policy trained exclusively on failure-vs-failure DPO pairs that autonomously generates high-reward, near-success hard negatives without external supervision.
  • Co-evolving framework: alternating optimization creates an implicit arms race — the failure agent progressively hardens the negatives while the target agent learns to overcome them.
  • Richer preference dataset for the target: expert–target, expert–failure-agent, and cross-agent failure–failure pairs combined into a single DPO training set.
  • Empirical validation on three multi-turn benchmarks (WebShop, ScienceWorld, InterCodeSQL) across Llama-2-7B, Llama-2-13B, and Qwen3-4B.

Results

  • Llama-2-7B vs. ETO baseline (avg reward): 64.1 vs. 58.3 (+5.8 pp)
  • WebShop: 66.7 vs. 63.0 (+3.7 pp)
  • ScienceWorld Seen: 69.7 vs. 65.6 (+4.1 pp)
  • ScienceWorld Unseen: 62.0 vs. 55.5 (+6.5 pp)
  • InterCodeSQL: 53.8 vs. 49.4 (+4.4 pp)
  • Qwen3-4B vs. ETO: 66.3 vs. 59.5 (+6.8 pp avg); InterCodeSQL jumps from 58.8 → 69.1 (+10.3 pp)
  • Failure trajectory statistics: failure agent produces 9–17 pp more negative trajectories and 2–9 pp more hard negatives (reward > 0.6) than ETO across all benchmarks.
  • Diversity: pairwise embedding distances of generated failures substantially higher than ETO (mean and variance both larger).

Limitations

  • Single rollout per instruction is used due to cost of long multi-turn trajectories, limiting failure coverage.
  • Hard-negative threshold is set at 0.6; ideally 0.7–0.8 trajectories are desired but occur in <1% of self-generating agent outputs, revealing a ceiling on difficulty.
  • Maintaining two agents doubles training compute and memory; experiments required 8×H100-80GB GPUs.
  • Co-evolution relies on an initial SFT stage with expert data, so some annotation overhead remains.

Relevance to Harnesses / Meta-Harnesses

The co-evolving agents framework is structurally a meta-harness: an outer training loop that orchestrates two specialized sub-agents, controls the flow of trajectories between them, and regulates when each agent trains — all without human intervention. The failure agent plays the role of a data-quality harness, continuously transforming raw interaction noise into structured, high-signal training pairs that feed the main agent's learning pipeline. This is analogous to meta-harness patterns where a coordinator module curates or filters outputs from sub-agents before passing them downstream. The alternating co-evolution schedule is itself a harness-level control structure — a designed protocol governing agent interaction order and data routing rather than any single agent's policy — directly relevant to researchers designing automated, iterative multi-agent orchestration systems.