Skip to content

Confucius Code Agent: Scalable Agent Scaffolding for Real-World Codebases

🕒 Published (v1): 2025-12-11 08:05 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Confucius Code Agent (CCA) is a software engineering agent built on the Confucius SDK, which structures agent design around three axes—Agent Experience (AX), User Experience (UX), and Developer Experience (DX)—with a meta-agent that automates the build-test-improve loop for agent configuration. CCA achieves 59% Resolve@1 on SWE-Bench-Pro with GPT-5.2, surpassing both research and proprietary commercial scaffolds under identical model and environment conditions. The key finding is that scaffolding quality can outweigh backbone model capability.

Problem

Existing coding agents either prioritize transparency (research-grade, poorly scalable) or performance (production-grade, opaque and inextensible). Both suffer from flat interaction histories, heuristic prompt engineering, tightly coupled tool pipelines, and static agent configurations requiring manual revision—none of which scale to long-horizon, multi-file enterprise codebases.

Method

CCA is built on the Confucius SDK with four mechanisms: 1. Hierarchical working memory + adaptive context compression: A file-system-based memory tree (session/entry/runnable scopes) backed by an Architect Agent that triggers structured summarization when context approaches thresholds, preserving goals/decisions/TODOs while maintaining a rolling recent window. 2. Note-taking agent: Asynchronously distills session trajectories into persistent Markdown notes—including hindsight notes on failure modes—stored in a typed file-system tree, enabling cross-session retrieval. 3. Extensions: Typed, composable modules registered as ordered callbacks (e.g., on_input_messages, on_llm_output) that handle output parsing, prompt shaping, tool invocation, and result summarization, cleanly separating orchestration from capabilities. 4. Meta-agent (build-test-improve loop): An agent that synthesizes orchestrator configs, tool wiring, and prompts from a high-level task description, then iteratively evaluates and refines them against a production test set—CCA itself is the output of this loop.

The orchestrator is a minimal LLM-call loop with native tool-use and XML-tag fallback, iteration-capped but agent-terminated.

Key Contributions

  • Confucius SDK: principled AX/UX/DX-decomposed agent development platform with modular extensions
  • Meta-agent that automates agent scaffold construction and refinement via evaluation-driven iteration
  • Hierarchical working memory + context compression enabling long-horizon reasoning without hard truncation
  • Persistent cross-session note-taking with hindsight failure capture
  • 59% Resolve@1 on SWE-Bench-Pro (GPT-5.2 backbone), surpassing OpenAI's proprietary scaffold; 74.6% on SWE-Bench-Verified (Claude 4 Sonnet), exceeding OpenHands

Results

  • SWE-Bench-Pro (731 tasks):
  • Claude 4 Sonnet: CCA 45.5% vs SWE-Agent 42.7%
  • Claude 4.5 Sonnet: CCA 52.7% vs Live-SWE-Agent 45.8%
  • Claude 4.5 Opus: CCA 54.3% vs Anthropic proprietary 52.0%
  • GPT-5.2: CCA 59.0% vs OpenAI proprietary 56.4%
  • Ablation (100-task subset): Context management adds +6.6 pp (Claude 4 Sonnet: 42.0→48.6); advanced tool use adds ~1.6 pp over simple tool use with same context management
  • Note-taking (151 tasks, Claude 4.5 Sonnet): Run 2 (with notes) vs Run 1: resolve rate 54.4% vs 53.0%, turns 61 vs 64, token cost 93k vs 104k
  • SWE-Bench-Verified: CCA (Claude 4 Sonnet) 74.6% vs OpenHands 72.8% vs mini-SWE-Agent (Claude 4.5 Sonnet) 70.6%
  • Multi-file robustness: Resolve@1 degrades moderately from 57.8% (1–2 files) to 44.4% (10+ files)

Limitations

  • Note-taking improvement is modest (+1.4 pp), and notes were only generated for 151/731 instances where meaningful insight could be distilled
  • Multi-file tasks show performance degradation; cumulative localization uncertainty and compounding diffs are unsolved
  • Cannot benchmark against Claude Code due to incompatibility with containerized SWE-rex evaluation environments
  • Meta-agent's learned tool conventions are not fully described or publicly auditable; refinement relies on a proprietary internal test set
  • Results depend on backbone model availability; CCA is not model-agnostic in practice (tested only on Claude 4.x/GPT-5.2 series)

Relevance to Harnesses / Meta-Harnesses

CCA is a direct exemplar of the meta-harness paradigm: the Confucius SDK is itself an agent development platform, and the meta-agent automates the construction and refinement of agent harnesses through an evaluation-driven loop rather than manual prompt engineering. The AX/UX/DX decomposition is a design framework for harness architecture—separating concerns across audience (model, human, developer) rather than conflating them in a single scaffold. The ablation evidence (scaffolding closes model-tier gaps) quantifies the leverage of harness design relative to model capability, directly relevant to anyone building or evaluating meta-harness systems. The note-taking and context compression mechanisms are reusable harness primitives applicable beyond coding agents.