Skip to content

Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Graham Neubig (CMU); agents handling underspecified SWE tasks with clarification

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Ambig-SWE extends SWE-Bench Verified with synthetically underspecified GitHub issues and an interactive evaluation framework where agents can query a GPT-4o user proxy holding the full specification. The benchmark evaluates three capacities: detecting underspecificity, asking targeted clarification questions, and leveraging interaction to improve task resolution. Interaction boosts resolve rates by up to 74% over non-interactive underspecified baselines, but models default to non-interactive behavior and struggle to reliably distinguish well-specified from underspecified inputs.

Problem

Current LLM agents processing underspecified instructions make unwarranted assumptions rather than asking clarifying questions, leading to misaligned solutions, safety risks from tool misuse, and wasted compute. Prior work on underspecificity studied single missing details; real agentic tasks involve multiple interdependent specification gaps (file locations, design decisions, constraints) that emerge across a multi-step trajectory. No evaluation framework existed for measuring agent interaction capability in realistic software engineering settings.

Method

The authors generate underspecified variants of all 500 SWE-Bench Verified issues using GPT-4o, instructed to preserve domain terminology while aggressively removing detailed content (code snippets, error messages, reproduction steps). Three evaluation settings are compared: Full (original issue, no interaction), Hidden (underspecified issue, no interaction), and Interaction (underspecified issue + GPT-4o user proxy with full specification). The proxy answers only from information explicitly in the full issue and responds "I don't have that information" for queried details not present, preventing hallucination. Agents run inside the OpenHands framework with Bash/Python sandbox access. Detection ability (RQ2) is measured by presenting each issue randomly as Full or Hidden and tracking when models choose to interact under three prompt-engineering conditions (Neutral / Moderate Encouragement / Strong Encouragement). Question quality (RQ3, partially shown) analyzes whether questions are actionable and sufficiently specific. Models evaluated: Claude Sonnet 4, Claude Sonnet 3.5, Claude Haiku 3.5, Llama 3.1 70B-Instruct, Deepseek-v2, Qwen 3 Coder 480B.

Key Contributions

  • Ambig-SWE benchmark: 500-issue interactive extension of SWE-Bench Verified with paired full/underspecified issues and a simulated user proxy, enabling causal measurement of interaction impact.
  • Three-capacity decomposition: Separates underspecificity handling into detection, question generation, and information integration—each evaluated independently to enable targeted model improvements.
  • Empirical interaction analysis: Demonstrates that compulsory interaction yields statistically significant resolve-rate gains over the Hidden setting for all models, and characterizes how navigational vs. informational queries affect performance differently across model families.
  • Prompt sensitivity study: Shows that detection behavior is highly sensitive to prompt phrasing but that prompt engineering alone cannot substitute for dedicated training.

Results

  • Interaction improves resolve rates by up to 74% over the Hidden (non-interactive underspecified) setting.
  • Claude Sonnet 4 recovers 89% of full-specification performance in the Interaction setting; Claude Sonnet 3.5 and Haiku 3.5 recover up to 80%; Deepseek-v2 59%; Llama 3.1 70B 54%.
  • Underspecificity detection (RQ2, best prompt per model): Claude Sonnet 4 achieves 89% accuracy (FPR 0.03, FNR 0.18) under Strong Encouragement; Claude Sonnet 3.5 84% under Moderate Encouragement; all others ≤63%.
  • Qwen 3 Coder achieves 100% FNR (never interacts) across all prompt conditions, reaching only chance-level 50% detection accuracy despite strong SWE-Bench task performance.
  • Navigational information (file paths) improves resolve rates for most models (e.g., Claude Sonnet 4: 60.8% → 67.2%; Claude Sonnet 3.5: 37.9% → 59.5%); Qwen 3 Coder uniquely degrades when provided file paths (55.4% → 52.4%), likely due to rigid re-exploration behavior.
  • Interaction increases step count without reducing it: Claude Sonnet 4 averages 65 steps (Hidden) vs. 75 steps (Interaction), indicating no efficiency gains from current interaction patterns.

Limitations

  • Underspecified issues are synthetically generated by GPT-4o rather than drawn from naturally occurring underspecified reports; distributional analysis shows generated issues lack conversational fragments, external links, and partial reproduction steps common in real reports.
  • The user proxy (GPT-4o) approximates but does not replicate real user behavior; it cannot hallucinate, which makes the setting more favorable than real-world interaction.
  • Evaluation is restricted to Python repositories in SWE-Bench; generalizability to other languages and non-GitHub task formats is untested.
  • Interaction is compelled via prompt instruction in RQ1, which does not reflect organic agent behavior; the gap between prompted and autonomous interaction represents a confound.
  • Claude Sonnet 4 is evaluated on only 100/500 Hidden instances due to cost, though results remain statistically significant.
  • The paper does not evaluate whether training on interactive trajectories would close the gap, leaving the training prescription empirically unverified.

Relevance to Agentic AI / LLM Agents

Ambig-SWE directly targets a critical failure mode of deployed LLM agents: proceeding with incomplete specifications rather than engaging in targeted clarification dialogue. The benchmark operationalizes underspecificity at the agentic-trajectory level—where multiple interdependent gaps accumulate across tool calls—rather than at the single-turn level studied by prior work, making it more representative of production agentic pipelines. The finding that models default to non-interactive behavior even under severe underspecification, and that prompt engineering cannot reliably fix this, implies that interactivity must be a first-class training objective rather than a prompting afterthought. The three-capacity decomposition (detect → ask → integrate) provides a diagnostic framework that agent designers can use to isolate which component is limiting system performance.