Skip to content

Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improve

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LessonL is a multi-agent framework where heterogeneous code LLMs iteratively share "lessons"—distilled knowledge about why solutions succeed or fail—to collectively improve performance on code optimization and generation. The core innovation is a solicitation–banking–selection mechanism that generates, stores, and filters lessons across rounds. A team of small open-source models using LessonL matches or outperforms GPT-4o and OpenAI o3 on code optimization benchmarks.

Problem

No single LLM dominates all coding subtasks: complementary strengths are distributed across models at fine granularity (e.g., Qwen7B beats GPT-4o by \(2.5\times\) on geometry problems; Deepseek7B beats GPT-4o by \(1.6\times\) on histograms). Existing multi-agent frameworks either require pre-assigned roles (which presupposes knowing each agent's strengths) or aggregate raw solutions without transferring reasoning knowledge between agents. Neither paradigm allows agents to learn from each other's successes and failures during a single inference run.

Method

LessonL operates in iterative rounds over a team of \(n\) agents:

  1. Solicitation: After each agent generates or updates a solution, it is prompted to produce a lesson—an explanation of what worked or failed—conditioned on the scenario outcome (speedup, slowdown, functional error, or syntax error) plus compiler/profiler feedback. Functional-incorrectness lessons deliberately omit test cases to encourage generalization.

  2. Banking: All lessons are deposited into a shared lesson bank.

  3. Selection: At round \(t\), \(k\) lessons are selected via a two-part rule: the top \(\lceil k/2 \rceil\) lessons by adjusted speedup score, and the top \(\lfloor k/2 \rfloor\) by cosine similarity to the original code (using CodeBERT embeddings). Negative lessons (failures) remain in the pool and are reachable via the relevance ranking.

  4. Effectiveness adjustment: Each lesson \(z\) has an adjustment factor \(f = c/n\), where \(c\) accumulates \(1+\epsilon\) each time applying \(z\) to an agent produces speedup \(> s_z\) and \(1-\epsilon\) otherwise. Lessons are ranked by \(s \times f\) rather than raw \(s\).

  5. Best solution returned after \(T\) rounds. The framework adapts to code generation by replacing speedup-based grading with pass-rate grading.

Key Contributions

  • Empirical demonstration that LLM complementarity exists at fine task-category granularity, not just aggregate benchmark level.
  • The LessonL framework: lesson solicitation, banking, and mixed speedup+relevance selection with dynamic effectiveness adjustment.
  • A team of three small open-source models (Deepseek7B + Qwen7B + Qwen14B) via LessonL outperforms GPT-4o (much larger) and matches/beats OpenAI o3 on code optimization under comparable resource budgets.
  • Lessons are human-interpretable and reusable as educational material—a byproduct of the collaboration mechanism.
  • State-of-the-art results on ParEval, PolyBench (code optimization), and HumanEval/MBPP suites (code generation).

Results

Code optimization (ParEval serial mode, geometric mean speedup): - LessonL: \(2.16\pm0.11\) vs. GPT-4o: \(1.72\pm0.11\), OpenAI o3: \(2.21\pm0.16\), MoA: \(1.76\pm0.08\)

Code optimization (ParEval OpenMP mode): - LessonL: \(3.46\pm0.03\) vs. GPT-4o: \(2.93\pm0.30\), OpenAI o3: \(3.55\pm0.27\), MapCoder: \(3.43\pm0.17\)

Code optimization (PolyBench OpenMP): - LessonL: \(3.40\pm0.12\) vs. GPT-4o: \(2.73\pm0.21\), OpenAI o3: \(3.15\pm0.23\)

Code generation (HumanEval pass@1): - LessonL: 0.915 vs. MoA: 0.909, GPT-4o: 0.884

Ablation: Removing lessons entirely (variant 5) drops OpenMP speedup from \(3.46\) to \(3.01\); removing dynamic speedup adjustment drops it to \(3.28\).

Scaling: Performance improves from 1 agent (speedup 1.60) to 6 agents (2.21) on ParEval serial.

Limitations

  • Evaluated primarily on scientific/parallel computing benchmarks (ParEval, PolyBench); generalization to other domains is demonstrated but less extensively tested.
  • The lesson selection heuristic (half by speedup, half by relevance) is fixed and may not be optimal across all task types—ablation shows different modes favor different strategies.
  • Requires multiple LLM inference calls per round per agent, increasing token cost significantly over single-agent baselines.
  • Effectiveness adjustment requires re-running lessons across agents each round to update \(f\), adding coordination overhead.
  • No formal convergence analysis; the number of rounds \(T\) is a hyperparameter tuned empirically.

Relevance to Harnesses / Meta-Harnesses

LessonL is a textbook example of a meta-harness: it wraps multiple heterogeneous LLM agents in a shared loop with a centralized state store (the lesson bank) that accumulates and routes structured knowledge between agents across rounds. The solicitation–banking–selection mechanism is directly analogous to a harness's dispatch–collect–filter pipeline, with the lesson bank acting as the inter-agent shared memory that a meta-harness manages. The dynamic effectiveness adjustment—updating lesson weights based on downstream agent outcomes—is a form of online feedback that meta-harnesses could adopt to route tasks away from consistently underperforming agents. For anyone building harnesses that coordinate specialist LLMs, LessonL's lesson format (scenario + explanation, graded by objective signal) provides a concrete, reusable pattern for structuring inter-agent communication that is more compact than passing full solution artifacts.