Skip to content

Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improve

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LessonL is a multi-agent framework where LLM agents collaboratively solve code optimization tasks by generating, banking, and selectively sharing "lessons"—concise natural-language explanations of why a solution succeeded or failed. Through iterative lesson solicitation and a prioritized lesson-selection mechanism, a team of small open-source LLMs (7B–14B parameters) surpasses GPT-4o on code optimization benchmarks at comparable resource cost.

Problem

No single LLM dominates across all coding subtasks—different models excel at different problem categories (e.g., Qwen7B outperforms GPT-4o by 2.5× on geometry problems). Existing multi-agent approaches either pre-assign fixed roles (planner, coder, debugger) or aggregate independent proposals; neither enables agents to learn from each other's successes and failures dynamically without knowing their complementary strengths a priori.

Method

LessonL implements a solicitation–banking–selection loop over T rounds: 1. Solicitation: After each agent generates/updates code, it is prompted to produce a lesson—explaining speedup gains, slowdowns, functional errors, or syntax errors—paired with the original and modified code and measured runtime feedback. 2. Banking: All n lessons per round are deposited into a shared lesson bank. 3. Selection: At most k lessons are fed to agents in the next round. The top ⌈k/2⌉ are chosen by effective speedup (adjusted dynamically by an effectiveness factor f = c/n tracking whether the lesson continued to help across agents); the top ⌊k/2⌋ by cosine similarity to the current code (via CodeBERT embeddings), mimicking RAG. 4. Effectiveness adjustment: A correction variable accumulates whether each lesson application outperformed or underperformed its original speedup, modulating its selection priority across rounds.

Agents are heterogeneous open-source models (Deepseek-7B, Qwen-7B, Qwen-14B); no role pre-assignment is required.

Key Contributions

  • Identification that LLM complementarity exists at fine-grained problem-category level, not just aggregate benchmark level.
  • The lesson-based collaboration paradigm: concise, interpretable knowledge units passed between agents rather than full code artifacts.
  • The solicitation–banking–selection mechanism with dynamic effectiveness adjustment.
  • Empirical demonstration that 3 small LLMs (~7–14B) with LessonL outperform GPT-4o on code optimization under similar resource budgets.
  • State-of-the-art results on both code optimization (ParEval, PolyBench) and code generation (HumanEval, MBPP) benchmarks.

Results

Code optimization (ParEval): - LessonL serial speedup: 2.16× vs. GPT-4o 1.72×, MapCoder 1.85×, MoA 1.76×, OpenAI o3 2.21× - LessonL OpenMP speedup: 3.46× vs. GPT-4o 2.93×, MapCoder 3.43×, OpenAI o3 3.55× - LessonL correctness on ParEval: 0.91 (serial), 0.86 (OpenMP), highest among open-source multi-agent methods

Code optimization (PolyBench): - LessonL serial speedup: 1.32× vs. GPT-4o 1.12×, MapCoder 1.17× - LessonL OpenMP speedup: 3.40× vs. GPT-4o 2.73×, MapCoder 2.23×

Code generation (pass@1): - HumanEval: 0.915 (best); HumanEval+: 0.878 (best); MBPP: 0.899 (best); MBPP+: 0.765 (tied with MoA)

Scaling: Performance improves monotonically from 1→6 agents, with biggest gains at 3 agents (1.60→2.12 serial speedup); diminishing returns beyond 3.

Cost: LessonL lies on the Pareto front of speedup vs. monetary/FLOP cost; MapCoder and MoA are least cost-effective.

Limitations

  • Lesson solicitation adds latency before first token, degrading time-to-first-response user experience.
  • Applied only to function-level code snippets; not yet validated on repository-level tasks (e.g., SWE-bench).
  • Some problems remain unsolvable even with collaboration when small LLMs lack the requisite low-level system knowledge (e.g., prefix-sum example); external knowledge integration is an open problem.
  • Effectiveness adjustment and selection heuristics involve several hyperparameters (k, ε, T) that may need tuning per task domain.
  • Cost estimates for closed-source models involve educated guesses about model size and pricing, which are time-sensitive.

Relevance to Agentic AI / LLM Agents

LessonL is a concrete instantiation of peer learning among LLM agents: rather than role specialization or voting aggregation, agents communicate via semantically rich, reusable lessons—a mechanism closer to how human expert teams iteratively refine shared knowledge. The solicitation–banking–selection design is a generalizable pattern for any multi-agent system where iterative refinement benefits from compact, retrievable inter-agent signals beyond raw outputs. The finding that small heterogeneous agent teams can outperform a single large model under equal compute is directly relevant to cost-efficient agentic deployments. The extension to code generation also suggests the lesson paradigm is task-agnostic, making it applicable beyond optimization to broader agentic coding pipelines.