U2F: Encouraging SWE-Agent to Seize Novelty without Losing Feasibility

🕒 Published (v1): 2025-11-05 14:46 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

U2F (Unknown Unknowns to Functional solutions) is a multi-agent scaffolding framework for software engineering agents that systematically surfaces "Unknown Unknowns" — solution pathways absent from initial problem formulations. It orchestrates three specialized agents (Discovery, Exploration, Integration) augmented by cross-domain analogical reasoning, reverse thinking, and external validation. On 218 real-world software enabler stories, it achieves a 14% novelty gain and 51% semantic novelty increase while holding feasibility stable at 4.02/5.0.

Problem

Existing LLM-based SWE-Agents operate within predefined solution paradigms, excelling at "Known Knowns" and "Known Unknowns" while systematically missing "Unknown Unknowns" (UUs) — solution paths, architectural options, and capability opportunities absent from initial requirements. Current agents also suffer from technology stacking without rationale, requirement drift, and hallucinated proposals when attempting open-ended exploration.

Method

U2F implements a three-agent pipeline inspired by the Double Diamond design model:

1. Discovery Agent: takes an enabler story and proposed fix, produces a concise problem statement, a baseline solution, and a structured analysis of its blind spots and implicit assumptions. It remains on standby to trigger full resets if later stages expose a flawed premise.

2. Exploration Agent: navigates a three-dimensional cognitive space using:

3. Cross-domain analogical reasoning: \(P_{\text{software}} \xrightarrow{\text{abstract}} P_{\text{general}} \xrightarrow{\text{map}} S_{\text{domain}}\) — abstracts the problem domain-independently, then maps solutions from unrelated fields (biology, economics, physics) back to software.
4. Reverse thinking: \(G_{\text{target}} \leftarrow P_1 \leftarrow P_2 \leftarrow \cdots \leftarrow P_{\text{minimal}}\) — backward reasoning from success criteria to expose hidden prerequisites and prune redundant components.

5. External validation: \(V(f) = F(f) + I(f) + C(f)\), assessing technical feasibility, implementation viability, and contextual appropriateness via live search.

6. Integration Agent: performs conflict mapping, solution refactoring, advantage attribution, and implementation planning; can demand re-exploration or trigger a Discovery-level reset.

A Search Augmentor (Google Search API) is woven throughout all three phases for on-demand grounding. A human-in-the-loop CLI allows expert override of optimization goals, taboo conditions, and reasoning paths at any stage (disabled during automated experiments).

Key Contributions

• First work to operationalize Unknown Unknowns discovery within LLM-based SWE-Agents, including formal definitions and a four-condition UU qualification criterion.
• Three complementary cognitive strategies (analogical reasoning, reverse thinking, external validation) operating at cognitive rather than prompt level.
• A curated dataset of 218 enabler stories derived from SWELancer tasks via three-model consensus filtering (Gemini 2.5-Flash, GPT-4o, Claude 3.7-Sonnet).
• Empirical validation using 15 domain experts plus an LLM judge (Qwen-3-32B) on a multidimensional 1–5 Likert rubric.

Results

• Novelty: +14% over baseline (human expert scores).
• Semantic novelty: +51% (\(1.5\times\)) improvement over baseline.
• Feasibility: maintained at 4.02/5.0 — no statistically significant degradation.
• Findings consistent between human panel (15 experts/students) and LLM judge (Qwen-3-32B).
• Baselines: zero-shot prompting, role-based prompting, and SEAP (Software Engineering Analysis Prompting — structured multi-step guidance combining requirements analysis, risk-driven development, domain-driven design, formal verification).

Limitations

• Human-in-the-loop interface was disabled during experiments for reproducibility; real-world performance with expert guidance is uncharacterized.
• Dataset constructed entirely from SAFe enabler stories derived from SWELancer; generalization to arbitrary software tasks is untested.
• Three-model consensus filtering explicitly excludes simple bugs and incomplete requirements, making the benchmark unrepresentative of the full SWE task distribution.
• Evaluation relies on subjective Likert scales; no objective, executable correctness metric is reported.
• Computational cost and latency of the three-agent pipeline plus live search are not quantified.

Relevance to Harnesses / Meta-Harnesses

U2F is structurally a meta-harness: it wraps three specialized sub-agents in a phased orchestration layer with explicit feedback loops, backtracking signals, and a cross-cutting Search Augmentor that any agent can invoke on demand — exactly the scaffolding architecture that characterizes a harness above bare LLM calls. The framework demonstrates how agent control flow can be organized around cognitive phases rather than task decomposition, with the Discovery Agent serving as a stateful reset mechanism that any downstream agent can trigger. The human-in-the-loop interface acts as a harness-level control plane, mirroring how meta-harnesses expose configuration hooks to operators. For researchers building general-purpose agent harnesses, U2F offers a concrete instantiation of adaptive, feedback-driven orchestration with quantified outcome metrics.