Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents¶

🕒 Published (v1): 2026-03-27 09:56 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Current LLM coding agents are optimized for autonomous execution and fail to seek clarification when given underspecified instructions. This paper proposes a two-agent scaffold that decouples underspecification detection from code execution, achieving a 69.40% resolve rate on an underspecified variant of SWE-bench Verified—matching agents given fully-specified inputs. The key finding is that separating intent monitoring into a dedicated "Intent Agent" produces well-calibrated information-seeking behavior, querying users selectively rather than compulsively.

Problem¶

LLM coding agents deployed on real-world GitHub issues routinely receive underspecified task descriptions that omit crucial context (e.g., which file, what error). Existing agents either attempt execution blindly (degraded performance) or are prompted with hardcoded instructions to ask questions (brittle, overclarifying, instruction-following failures). No prior work evaluates independent, calibrated clarification-seeking in a dynamic multi-turn software engineering setting.

Method¶

The authors construct an underspecified variant of SWE-bench Verified (500 GitHub issues summarized by GPT-4o to remove critical details) and evaluate agents within the OpenHands framework (max 100 iterations, sandboxed code execution).

Two uncertainty-aware scaffolds are proposed:

UA-Single: A single agent is prompted at every turn to check for underspecification and, if detected, query the user—no hardcoded interaction requirement.
UA-Multi: A two-agent system where a Main Agent handles repository navigation and code editing, while a dedicated Intent Agent monitors the full state history each turn and outputs a binary decision (clarification needed / not needed). When the Intent Agent flags underspecification, the Main Agent's next action is constrained to a user query. Both agents share the same LLM backbone.

A GPT-5.1 user simulator with access to the fully-specified original issue answers agent queries, constrained to withhold only the missing context.

Baselines: Full (fully specified, no interaction), Hidden (underspecified, no interaction), Interactive Baseline (underspecified, hardcoded first-turn query forced).

Backbones evaluated: Claude Sonnet 4.5 (proprietary) and Kimi K2.6 (open-weight).

Key Contributions¶

An underspecification-aware multi-agent scaffold (UA-Multi) that decouples intent monitoring from code execution without modifying the standard SWE-bench task prompt.
Empirical evidence that frontier LLMs can exhibit calibrated clarification-seeking: higher ask rates on harder tasks, abstaining on already-resolvable issues.
Demonstration that hardcoded interactive prompts are brittle (Kimi K2.6 Interactive Baseline collapses to 47.20% due to instruction-following failures).
Analysis of query timing and distribution: UA-Multi with Claude Sonnet 4.5 averages 3.06 queries/task vs. 1.84 for UA-Single; Kimi K2.6 averages 8.71 queries/task, concentrated in the middle execution phase.

Results¶

UA-Multi resolve rate: 69.40% for both Claude Sonnet 4.5 and Kimi K2.6
Claude Sonnet 4.5:
Full: 70.80% → UA-Multi closes gap (p = 0.458 vs. Full; p = 0.621 vs. Interactive Baseline)
Hidden: 54.80%; UA-Single: 61.20%; UA-Multi: 69.40%
UA-Multi improvement over UA-Single: p < 0.001
On tasks where UA-Multi refrained from asking (N=156): 76.92% resolve rate, near-matching Hidden (77.56%)
On tasks where UA-Multi asked (N=344): 65.99% resolve rate, matching Full (66.57%)
Kimi K2.6:
Full: 72.80%; Hidden: 55.40%; Interactive Baseline: 47.20% (collapse)
UA-Single: 61.60%; UA-Multi: 69.40% (p < 0.001 over UA-Single; p < 0.05 gap to Full remains)
Calibration: UA-Multi (Claude) shows 9.28% higher ask rate for medium tasks vs. easy tasks; UA-Single shows only 2.43% increase; Kimi K2.6 UA-Single ask rate declines on harder tasks.

Limitations¶

LLM user simulator: GPT-5.1 simulator is unnaturally cooperative and may not reflect real human variance or reluctance.
Frontier-only generalization: Smaller open-weight models lack calibration for interactive underspecification; results may not transfer below frontier scale.
Domain scope: Findings are specific to SWE-bench Verified; do not extrapolate to security-critical or other high-stakes settings without further study.
Cost overhead: Multi-agent inference significantly increases financial and environmental cost; Appendix F documents the breakdown.
Backbone sensitivity: Kimi K2.6 exhibits higher query rates and poorer calibration (8.71 queries/task vs. 3.06 for Claude), suggesting calibrated behavior requires more than scaffold design alone—possibly targeted training.

Relevance to Harnesses / Meta-Harnesses¶

UA-Multi is a meta-harness pattern: it wraps an existing coding agent (OpenHands) with a supervisory Intent Agent that intercepts execution at every step and gates continuation on an uncertainty signal—structurally analogous to meta-harnesses that wrap base agents with monitoring, routing, or control layers. The binary-decision Intent Agent is a minimal but effective orchestration primitive—a per-turn, stateful controller that modifies the downstream agent's action space, which is directly composable into larger harness architectures. The finding that decoupling detection from execution outperforms both single-agent and hardcoded prompting validates the harness abstraction as a source of capability gain beyond model-level improvements. The calibration analysis (selective querying correlated with task difficulty) also informs harness design decisions around when to hand off control vs. proceed autonomously.