Skip to content

Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DEI (Diversity Empowered Intelligence) is a meta-module that sits atop existing SWE agent frameworks and selects the best candidate patch from a committee of heterogeneous agents via LLM-based code review and scoring. On SWE-Bench Lite, a committee of open-source agents with a maximum individual resolve rate of 27.3% reaches 34.3% under DEI; the best DEI group achieves 55.0%. The key insight is that diverse agents resolve largely non-overlapping issue sets, creating large oracle headroom that DEI partially closes.

Problem

Individual SWE agents exhibit complementary strengths: agents with nearly identical aggregate resolve rates (~26–31%) solve very different subsets of GitHub issues, leaving a large gap between their union (oracle) performance and their average performance. No prior framework systematically harvests this inter-agent diversity for patch selection.

Method

DEI is formalized as a meta-policy \(\pi_{\text{DEI}}\) that, given context \(c\), selects among \(N\) specialist agent policies \(\{\pi_1, \ldots, \pi_N\}\): $\(\pi_{\text{DEI}} = \max_{\pi} \mathbb{E}_{c \sim \rho}\left[\mathbb{E}_\tau\left[\sum_{t=0}^{T} R(s_t, a_t, c)\;\middle|\; c;\, \pi(c)\right]\right]\)$

The concrete implementation, DEI\(_{\text{BASE}}\), is a three-step LLM-as-judge pipeline: 1. Input construction — for each candidate patch, assemble (issue description, relevant code spans from Moatless Tools, code before patch, code after patch). 2. Explanation generation — the LLM produces five chained explanations in order: issue explanation → context explanation → location explanation → patch explanation → conflict detection. Each step is conditioned on prior ones. 3. Patch scoring — the LLM scores each patch 1–10 using rubrics (wrong modification location = severe deduction); scoring is repeated at temperature 1.2 and averaged; top-\(k\) candidates are selected.

Key Contributions

  • First systematic empirical characterization of intra-agent (multiple runs of one agent) and inter-agent (across distinct frameworks) diversity on SWE-Bench Lite, showing Union@10 exceeds Average@10 by \(\geq 2\times\) for multi-agent settings.
  • Introduction of DEI, a framework-agnostic meta-policy that wraps any set of SWE agents without modifying them.
  • DEI\(_{\text{BASE}}\)-Open (4 open-source agents) reaches 34.3% resolve rate, beating most closed-source single agents.
  • DEI\(_{\text{BASE}}\)-1 achieves 55.0% on SWE-Bench Lite, the highest leaderboard rank at time of submission.
  • Empirical finding: inter-agent diversity is more exploitable than intra-agent diversity; given a fixed candidate budget, choosing diverse agents outperforms multiple runs of the same agent.

Results

  • DEI\(_{\text{BASE}}\)-1 (top-2 agents): 55.0% resolve rate on SWE-Bench Lite (#1 leaderboard at cutoff).
  • DEI\(_{\text{BASE}}\)-2 (5 closed-source agents): 50.6% resolve rate.
  • DEI\(_{\text{BASE}}\)-Open (4 open-source agents): 34.3% vs. best individual resolve rate of 27.3% (+25% relative).
  • Oracle upper bound (Union@10, 10 diverse agents): 54.3% vs. average single-agent 26.6% — DEI\(_{\text{BASE}}\) closes roughly half the gap.
  • DEI@k consistently exceeds Average@k across all \(k\) and all four agent groupings (inter-agent and three intra-agent settings), though gains plateau and slightly reverse at large \(k\).

Limitations

  • DEI\(_{\text{BASE}}\) relies on relevant code spans from Moatless Tools regardless of the candidate patch source — a mismatch that may degrade scoring quality.
  • Performance plateaus or slightly drops at large \(k\); the LLM reviewer is not ideal for selecting among many candidates simultaneously.
  • The framework requires running multiple full SWE agents per issue, multiplying inference cost.
  • Evaluation is limited to SWE-Bench Lite (300 instances); generalization to harder or domain-shifted benchmarks is untested.
  • No mechanism for the meta-policy to request additional candidates or to route issues to specific agents proactively; selection is purely post-hoc.

Relevance to Harnesses / Meta-Harnesses

DEI is a direct instantiation of a meta-harness pattern: it orchestrates a committee of independently-running sub-agents, aggregates their outputs, and applies a separate judge layer to select the best result — without modifying any constituent agent. The architecture cleanly separates the harness (meta-policy + scoring pipeline) from the workers (SWE agents), making it a reusable wrapper composable with any future agent. The LLM-as-judge scoring pipeline with chained explanation generation is a concrete, transferable technique for output-selection stages in meta-harness designs targeting code or structured artifact generation tasks.