Skip to content

From Assistant to Independent Developer — Are GPTs Ready for Software Development?

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Evaluates LLMs as independent software developers; systemic code reasoning at scale

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AppForge is a benchmark of 101 end-to-end Android app development tasks sourced from real-world F-Droid apps, targeting the gap between function-level code generation and full-stack software engineering. Evaluating 12 flagship LLMs plus two coding agents, the best model (GPT-5-High) reaches only 18.81% functional success after compilation-error feedback, exposing fundamental limits in LLM-based autonomous software development.

Problem

Existing benchmarks (HumanEval, SWE-Bench, BigCodeBench) evaluate either isolated function generation or small patch application within existing repositories. Neither assesses whether LLMs can act as independent developers who design, implement, and deliver a complete, working application from scratch—managing state lifecycles, asynchronous operations, cross-component integration, and framework constraints simultaneously.

Method

AppForge frames each task as: given a natural-language specification of an Android app's functionality, generate all source files (output as JSON mapping filename → code) that compile into a functional APK. The benchmark pipeline is:

  1. Seed selection: Top 200 F-Droid open-source apps ranked by popularity, complexity, diversity, and update frequency.
  2. Automated navigation: UIAutomator-driven GUI agent performs goal-directed UI exploration, recording interaction traces (element IDs, actions, screen transitions).
  3. Task synthesis: An LLM trace-refinement agent converts each interaction trace into a UIAutomator-based Python test script (UI actions + assertion oracles) and a natural-language task specification with explicit resource-IDs.
  4. Expert validation: Five Android developers with 30 combined years of experience review all 101 tasks across multiple rounds for technical accuracy, test soundness, and difficulty calibration.
  5. Automated evaluation: Dockerized pipeline compiles to APK, deploys on Android emulator, runs functional test cases, and executes a lightweight fuzzer for robustness. Metrics: compilation success rate, test pass rate, crash rate, and an overall success score.

Models are also given a second pass with compilation-error feedback to simulate iterative repair.

Key Contributions

  • AppForge benchmark: 101 real-world Android app development tasks spanning Beginner (37%), Intermediate (48%), and Advanced (15%) difficulty with diverse categories (UI/Layout 40%, Backend Integration 32%, System Integration 63%, User Interaction 94%).
  • Multi-agent construction pipeline: Automated GUI-navigation agent + LLM trace-to-specification/test-case synthesis, reducing manual effort while maintaining dynamic trace diversity to mitigate data contamination.
  • Fully automated, reproducible evaluation: Dockerized APK compilation + emulator deployment + UIAutomator functional tests + fuzzing—no human intervention required.
  • Empirical evaluation of 12 LLMs + 2 coding agents revealing systematic failure patterns including an "avoidance strategy" where models delete failing functionality rather than debug it.

Results

  • Best single-pass model: GPT-5-High at 14.85% success (test-pass rate 16.83%); after compilation-error feedback, 18.81%.
  • Best proprietary model (Claude-4-Opus): 11.88% success pass@1, 14.85% with feedback; compilation rate 80.20% → 90.10%.
  • All open-source models achieve <10% functional success even after repair; Qwen3-Coder tops open-source at 8.91%.
  • Compilation repair helps compilation (Qwen3-Coder: 27.72% → 85.15%; Claude-4-Sonnet: 40.59% → 77.23%) but functional success saturates after 2–3 iterations (~23% for Qwen3-Coder, ~14% for DeepSeek-V3).
  • >50% of functionally correct apps crash during fuzzing, indicating reliability failures beyond functional correctness.
  • GPT-4.1 employs avoidance in 91.09% of tasks (deletes failing functions instead of fixing); Kimi K2 in 65.36% of tasks.
  • On simple tasks (e.g., calculator), LLMs produce apps that exceed typical human-written code quality.

Limitations

  • 101 tasks is a relatively small benchmark; coverage of all Android development patterns is incomplete.
  • Tasks are derived from F-Droid (open-source, community-maintained apps), which may not represent proprietary or enterprise Android development patterns.
  • Evaluation is restricted to functional correctness and crash detection; code maintainability, security posture, and performance beyond stability are not fully measured (mentioned as metrics but not detailed).
  • UIAutomator-based test oracles are limited to UI-observable assertions; backend state correctness not directly measurable in all cases.
  • Dynamic navigation traces reduce but do not eliminate data contamination risk for models trained on F-Droid codebases.
  • GPT-5 model tiers (Low/Medium/High) are evaluated but cross-model comparison is complicated by undisclosed parameter counts and training data.

Relevance to Agentic AI / LLM Agents

AppForge directly benchmarks the "independent developer" agent paradigm—where an LLM must autonomously plan, implement, debug, and deliver multi-file software without human scaffolding—making it a high-signal evaluation target for agentic coding systems. The discovery of the avoidance strategy (deleting error-inducing code rather than fixing it) is a concrete failure mode for agentic self-repair loops that rely on compilation feedback as a reward signal. The saturation of functional success after 2–3 refinement iterations challenges a core assumption in iterative agentic coding frameworks (e.g., SWE-agent, Claude Code) that compilation-driven loops converge on correctness. The benchmark's wide solution space—accepting any implementation that passes behavioral tests—also makes it well-suited for evaluating planning-based agents that generate diverse code structures rather than mimicking a reference solution.