Skip to content

Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CURE is a reinforcement learning framework that co-evolves a single LLM's coding and unit test generation capabilities without requiring ground-truth code solutions. It derives individual-level rewards for unit tests from a theoretically grounded precision metric (µ), enabling the coder and tester to mutually supervise each other. The resulting models outperform same-size baselines on both code generation and unit test accuracy, and can serve as reward models to bootstrap RL on new base models.

Problem

Training LLM unit test generators has relied on ground-truth code solutions for supervision, limiting scalability and data diversity. Meanwhile, one-shot code generation accuracy alone is insufficient for agentic/test-time-scaling settings, which require accurate unit tests to enable self-checking and self-correction. No prior work jointly optimizes code generation and unit test generation without labeled code supervision.

Method

CURE frames co-evolution as a joint RL problem over a single shared policy acting as both coder and unit tester. For each task, the policy generates n code solutions and m unit tests, which are cross-executed to produce a binary evaluation matrix B* ∈ {0,1}^{n×(m+t_q)}, where t_q ground-truth unit tests anchor code rewards.

Code reward (Eq. 3): number of ground-truth unit tests passed by each solution.

Unit test reward (Eq. 4): an estimator of µ = p_u(1 − p_{01}) − (1 − p_u)p_{00}, derived from Theorem 3.1, which proves that reward precision → 1 as m → ∞ iff µ > 0. Concretely, R*_{u_k} is positive and proportional to how many incorrect solutions a test correctly rejects, and negative if the test fails any correct solution. This discourages trivially permissive tests and encourages discriminative power.

Both objectives are optimized with a GRPO-style clipped policy gradient plus KL regularization. For long-CoT base models (Qwen3-4B), a response-length-guided reward transformation penalizes overly long unit test responses while preserving the sign of original rewards, improving inference efficiency.

Key Contributions

  • CURE framework: RL co-evolution of coder and unit tester without any ground-truth code supervision, with a theoretically derived per-test reward (µ-estimator).
  • Response-length-guided reward transformation for long-CoT models, reducing unit test generation length to 64.8% of baseline while preserving accuracy.
  • CURE-4B/7B/14B models demonstrating consistent improvements over Qwen2.5-Instruct and Qwen2.5-Coder baselines across five benchmarks.
  • CURE as a reward model: CURE-generated unit tests bootstrap RL on separate base models with performance competitive to ground-truth-supervised RL.
  • Cross-model applicability: CURE-4B used as unit tester for GPT-4o-mini and GPT-4.1-mini improves their BoN accuracy while reducing API cost.

Results

  • CURE-7B/14B surpass respective Qwen2.5-Instruct bases by +24.4% unit test accuracy, +4.5% one-shot code accuracy, +5.1% BoN accuracy (16 codes × 16 unit tests), averaged across five benchmarks.
  • CURE-14B outperforms Qwen2.5-14B-Coder-Instruct (SFT baseline) on all three metrics.
  • CURE-4B (long-CoT) outperforms Qwen3-4B on unit test accuracy, code accuracy, and BoN accuracy while reducing unit test response length by 35.2% (to 64.8% of original).
  • Applied to agentic pipelines (MPSC, AlphaCodium, S): +6.2% average accuracy* over Qwen2.5-14B-Instruct.
  • Agentic unit test generation: +25.1% average accuracy over base model.
  • CURE-4B as external unit tester for GPT-4o-mini: +7.0% over GPT-4o one-shot at lower cost; GPT-4.1-mini BoN: +4.7% average over one-shot baseline.
  • CURE-4B as reward model for RL on Qwen2.5-14B-Instruct: performance matches RL trained with labeled ground-truth unit tests.
  • Ablation: the theoretically derived µ reward constrains p_{01}/p_{00} to 30.1%/10.6% vs. 40.5%/15.8% for a simpler pu-only reward.

Limitations

  • Currently restricted to Python competition-style tasks evaluated via stdin/stdout unit tests; functional tests and multi-language support not yet addressed.
  • Still depends on a small number of ground-truth unit tests during RL training (to anchor code correctness labels); fully unsupervised training remains future work.
  • Gains from CURE on long-CoT models are more modest than on standard base models, consistent with the observation that CoT reasoning already captures much of the BoN benefit.
  • Evaluation focuses on competitive programming datasets (LiveCodeBench, CodeContests, CodeForces, MBPP, LiveBench); generalization to real-world software engineering tasks (e.g., SWE-bench) is untested.

Relevance to Agentic AI / LLM Agents

CURE directly addresses a core bottleneck in agentic coding pipelines: the need for reliable execution-based reward signals (unit tests) without expensive human annotation. By co-evolving test generation with code generation via RL, CURE enables self-checking and self-correction loops that are central to agentic workflows (Best-of-N, iterative debugging, AlphaCodium-style refinement). The demonstrated +25.1% gain in agentic unit test generation and compatibility with MPSC/S*/AlphaCodium pipelines show it is a drop-in improvement for existing multi-step coding agents. More broadly, CURE's approach of training one model component to supervise another via interaction outcomes—without labeled data—is a generalizable pattern for bootstrapping reward models in agentic RL settings.