Skip to content

Solvita: Enhancing Large Language Models for Competitive Programming via Agentic Evolution

🕒 Published (v1): 2026-05-14 18:15 UTC · Source: Arxiv · link

Why this paper was selected

Solvita: stateful self-evolving agent accumulates experience for competitive programming

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Solvita is a multi-agent framework for competitive programming that achieves continuous learning without LLM weight updates by pairing four specialized agents (Planner, Solver, Oracle, Hacker) each with a trainable, graph-structured knowledge network updated via reinforcement learning signals. Unlike stateless pipelines, failure signals propagate across all agents, allowing accumulated problem-solving experience to reshape future routing. On CodeContests with GPT-5.4, Solvita lifts pass@1 from 40.0% to 82.4%—nearly double the single-pass baseline.

Problem

Existing multi-agent coding frameworks (AlphaCodium, MapCoder) are fundamentally stateless: each problem is solved from scratch and experience from prior successes or failures is discarded. RAG-based memory variants treat retrieval as static similarity lookup and inject raw text into prompts without altering the underlying reasoning procedure. This prevents LLM-based systems from accumulating the transferable experience that human competitive programmers rely on to improve.

Method

Solvita organizes problem-solving into a closed loop of four specialized agents:

  1. Planner — formalizes the problem into a mathematical specification, predicts algorithmic tags, and proposes a strategy sketch with complexity estimate.
  2. Solver — implements the strategy in C++, using patch-based repair (SEARCH/REPLACE edit blocks) rather than full regeneration on failure; backed by a three-layer heterogeneous directed graph \(G = (V_Q \cup V_M \cup V_S,\; E_{QM} \cup E_{MS})\) linking problem queries (Q), metacognitive decompositions (M), and reusable skill templates (S). Skill selection scores aggregate over two-hop paths: $\(\rho(s \mid q_\text{new}) = \sum_{q_i,\, m_j:\; q_i \to m_j \to s \in G} \mathrm{Sim}(q_\text{new}, q_i) \cdot w_{qm}^{(i,j)} \cdot w_{ms}^{(j,s)}\)$
  3. Oracle — generates certified internal test suites using testlib (generator, validator, custom checker); accepts an artifact only when certification ratio \(\rho = N_\text{cert}/N_\text{target} \geq \tau\) and output/input sets are non-empty.
  4. Hacker — mounts adversarial attacks via three routes (semantic corner-case, maximum-bound stress, lattice-based hash-collision); bugs discovered propagate via shared event bus to all four knowledge networks.

Each agent's knowledge network is trained via a contextual-bandit policy; edge weights update by REINFORCE using counterfactual rewards (e.g., \(\Delta R = R_\text{with} - R_\text{without}\) for the Solver). The graph grows dynamically: contrastive M nodes are added precisely where the agent currently struggles. A 30k-problem corpus is assembled from Codeforces, AtCoder, Aizu, etc., then filtered through four sequential stages (completeness → tag load-balancing → embedding deduplication with text-embedding-3-large → difficulty pruning) down to 8,017 training problems.

Key Contributions

  • Agentic evolution without weight updates: trainable graph-structured knowledge networks enable continuous improvement of frozen LLM backbones via RL edge-weight updates.
  • Patch-based repair with regression gating: the Solver emits SEARCH/REPLACE blocks accepted only if all prior regression tests still pass, preserving correctness incrementally.
  • Certified internal supervision (Oracle): four-stage pipeline with a formal acceptance gate (Eq. 3) ensures test artifacts meet quality thresholds before use.
  • Cascading adversarial testing (Hacker): three-route attack cascade with a structured reward (Eq. 4) weighted on break rate and severity, and cross-agent event propagation.
  • Cross-agent failure propagation: a shared event bus causes Hacker discoveries to write contrastive entries into Planner, Solver, and Oracle networks simultaneously.
  • Curated training corpus: 8,017 competitive-programming problems after a deliberate four-stage filtering pipeline controlling completeness, tag balance, redundancy, and difficulty floor.

Results

  • CodeContests (CC), GPT-5.4: Solvita 82.42% pass@1 vs. single-pass 40.00%, Codex CLI 81.82%, Claude Code 70.91%, AlphaCodium 60.61%, MapCoder 57.58%.
  • APPS, GPT-5.4: Solvita 67.70% vs. single-pass 37.90%, Codex CLI 67.10%, Claude Code 60.40%.
  • AetherCode (AC), GPT-5.4: Solvita 49.25% vs. single-pass 18.00%, Codex CLI 48.50%, Claude Code 42.75%.
  • Best in 14 of 15 backbone–benchmark cells across GPT-5.4, Claude Opus 4.6, Qwen3.6, DeepSeek V4 Pro, Grok; only exception is AetherCode under Claude Opus 4.6.
  • Token footprint comparable to open-source frameworks, substantially lower than commercial CLI agents.
  • Ablation: multi-agent scaffold without trained networks already closes most of the gap; Solver knowledge network gives the largest single-component gain; gains accumulate monotonically over the 5,318-problem training trajectory (checkpoints at 1.5k, 3k, 4.5k).

Limitations

  • The LLM backbone is frozen; Solvita's improvements are entirely confined to the knowledge network layer—underlying reasoning capacity is not increased.
  • Knowledge network cold-start requires a curated corpus of 8,017 problems before online learning begins; bootstrapping for new domains is non-trivial.
  • Evaluation is limited to competitive programming; generalization to other structured-reasoning domains is not demonstrated.
  • The one exception cell (AetherCode, Claude Opus 4.6) suggests potential backbone-dataset interactions that remain unexplained.
  • Hacker budgets (max 3 hack rounds per candidate) are fixed heuristics; sensitivity analysis is deferred to appendices and may not transfer to other problem distributions.
  • The corpus is scraped from a fixed set of platforms; platform API or policy changes would disrupt data collection.

Relevance to Harnesses / Meta-Harnesses

Solvita is a concrete instance of a meta-harness architecture: it wraps a frozen base model in a structured orchestration layer (four specialized agents + shared event bus) and uses outcome signals from within that layer as the training signal for learnable routing modules, closing the harness's own feedback loop without touching the underlying LLM. The per-agent knowledge networks functioning as learned dispatchers—routing queries to strategies based on accumulated pass/fail history—are directly analogous to the "learned meta-controller" pattern relevant to harness design. The cross-agent event bus, which propagates Hacker discoveries as contrastive training entries into Planner/Solver/Oracle networks, demonstrates how a harness can maintain global coherence across independently specialized sub-harnesses. For researchers building meta-harnesses, Solvita's contrastive REINFORCE update on graph edge weights (rather than LLM parameters) is a practically deployable mechanism for making a harness that learns from its own operational history.