UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development

🕒 Published (v1): 2026-07-02 13:56 UTC · Source: Arxiv · link

Why this paper was selected

Uncertainty-aware role collaboration in multi-agent software dev; addresses reliability gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UA-ChatDev extends the ChatDev multi-agent software development framework by injecting an uncertainty quantification (UQ) module between agent subtasks, using token-level log-probabilities to detect low-confidence outputs and triggering retrieval-augmented verification before errors propagate downstream. On the SRDD benchmark it raises Overall Quality from 0.395 (ChatDev) to 0.649 (Qwen 2.5 backbone), with particularly large gains in executability. The approach is model-agnostic, demonstrated on both Gemma 2 9B and Qwen2.5-Coder 7B.

Problem

Existing LLM-based multi-agent software development frameworks (ChatDev, MetaGPT, GPT-Engineer) treat every intermediate agent output as equally reliable. This assumption enables hallucination propagation: a misinterpreted requirement, hallucinated API, or incomplete code artifact produced in an early phase is silently forwarded to downstream agents, compounding errors and degrading final software quality. No prior framework explicitly models or gates on output confidence at agent-to-agent handoff points.

Method

UA-ChatDev wraps the ChatDev pipeline (Design → Coding → Testing phases, each with multiple subtasks) with a lightweight UQ module that intercepts inter-subtask communications.

Uncertainty estimation. Given a response \(r_i\) comprising \(n\) tokens with raw logits \(\{s_k\}\), per-token log-probabilities are computed as: $\(p_k = \log \frac{\exp(s_k)}{\sum_v \exp(s_v)}\)$ Temperature-scaled softmax (\(t = 1.0\)) then yields probability estimates \(z_k\). Response uncertainty is: $\(u = -\log \operatorname{Avg}(z_1, \ldots, z_n)\)$ For single-token (yes/no) decisions, \(u = |p_1|\) directly. For proprietary APIs exposing log-probs natively, the same formula applies; for local HuggingFace models, only top-\(k\) log-probs per position are retained for efficiency.

Phase-aware threshold calibration. Rather than a global threshold, each subtask \(\phi\) has its own threshold \(\tau_\phi\) computed as the 90th percentile (\(\alpha = 0.10\)) of uncertainty values from historical responses whose downstream code passed the next subtask: $\(\tau_\phi = Q_{0.90}\{u : \text{response in subtask } \phi \text{ is correct}\}\)$ A global fallback \(\tau = 0.5\) is used when fewer than 50 correct historical responses exist.

Retrieval trigger. When \(u > \tau_\phi\), web retrieval is invoked to augment the agent's context before it proceeds; otherwise the standard chat-chain workflow continues unmodified.

Key Contributions

• UA-ChatDev framework: first integration of uncertainty quantification into agent-to-agent handoffs within a multi-agent software development pipeline.
• Lightweight single-inference UQ: token-level log-probability aggregation avoids costly multi-sample estimation; open-source LLM support via log-softmax over full vocabulary logits.
• Phase-aware threshold calibration: per-subtask 90th-percentile thresholds derived from historical correct runs, preventing mis-firing from response-length heterogeneity across subtasks.
• Model-agnostic validation: consistent improvements over ChatDev demonstrated with both Gemma 2 9B and Qwen2.5-Coder 7B on the SRDD 1,200-task benchmark.

Results

All results on the SRDD benchmark (1,200 tasks, 5 categories). Metrics: Completeness (C), Executability (E), Consistency (K, CodeBERT cosine similarity), Overall Quality (\(Q = C \times E \times K\)).

System	C	E	K	Q
GPT-Engineer	0.502	0.358	0.789	0.142
MetaGPT	0.483	0.415	0.760	0.152
ChatDev	0.560	0.880	0.802	0.395
UA-ChatDev (Gemma 2 9B)	0.767	0.840	0.929	0.596
UA-ChatDev (Qwen2.5 7B)	0.818	0.859	0.927	0.649

• Overall Quality improves +64% relative vs. ChatDev (0.649 vs. 0.395, Qwen 2.5).
• Completeness: 0.560 → 0.818 (+46%); Consistency: 0.802 → 0.927 (+16%) vs. ChatDev.
• Ablation: removing UQ drops Qwen 2.5 Executability from 0.859 → 0.779 and Overall Quality from 0.649 → 0.578.
• Communication analysis: Code Review dominates coding-phase interactions (59.73%); reviewer agents surface recurring execution-level defects (missing imports, unhandled cases, missing initialization) across multiple rounds.
• Cost: UA-ChatDev (Qwen 2.5) uses ~40,490 tokens and ~306 s vs. ChatDev's ~22,949 tokens and ~148 s — roughly 1.76× token overhead and 2× latency.

Limitations

• Computational overhead: ~1.76× more tokens and ~2× generation time compared to ChatDev; not suitable for latency-sensitive applications.
• Threshold calibration requires historical data: the 90th-percentile calibration needs ≥50 correct responses per subtask; otherwise falls back to a fixed global \(\tau = 0.5\) which may be poorly tuned.
• Open-source model log-prob extraction: full vocabulary logit retention is memory-intensive; only a top-\(k\) approximation is used, introducing approximation error in the uncertainty estimate.
• Single-inference UQ: avoids multi-sample sampling (e.g., Monte Carlo dropout, self-consistency) for efficiency, but may be less calibrated than ensemble-based methods.
• Retrieval quality not ablated: the benefit of web retrieval vs. other augmentation strategies (e.g., in-context example retrieval, tool use) is not separately evaluated.
• Evaluation limited to SRDD: all tasks are relatively small programs; generalization to large-codebase, real-world projects is unvalidated.

Relevance to Agentic AI / LLM Agents

UA-ChatDev addresses a fundamental reliability gap in multi-agent pipelines: the silent propagation of low-confidence intermediate outputs across agent boundaries—a failure mode that is architecturally distinct from single-agent hallucination and requires inter-agent intervention mechanisms. The token-level log-probability UQ approach is lightweight enough to deploy inline without retraining, making it a practical primitive for any role-based agent workflow, not just software development. The phase-aware threshold calibration strategy is a concrete operationalization of context-sensitive trust in agent handoffs, relevant to broader work on agent guardrails, selective verification, and confidence-gated tool use. This connects directly to open problems in agentic reliability: when should an agent ask for help, re-verify, or augment its context before acting?