Skip to content

D-GARA: A Dynamic Benchmarking Framework for GUI Agent Robustness in Real-World Anomalies

🕒 Published (v1): 2025-11-20 17:43 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

D-GARA is a dynamic benchmarking framework for evaluating Android GUI agent robustness against real-world anomalies (permission dialogs, crashes, network drops, etc.) injected live during task execution. It introduces semantic rule-based anomaly triggering, state-centric success validation, and a 152-task benchmark (D-GARA-152) across 8 apps. All tested SOTA agents degrade substantially under interruptions, with average SR drops exceeding 17.5%.

Problem

Existing GUI agent benchmarks are static and idealized—they evaluate agents on fixed screenshot sequences without unpredictable system events. Static anomaly injection (e.g., GUI-Robust's screenshot overlays) only simulates shallow, reversible interruptions that don't alter execution trajectories. This masks agent fragility: models performing well on static benchmarks fail badly in live Android environments where permission dialogs, app crashes, and network alerts reroute control flow entirely.

Method

D-GARA wraps an Android emulator with two runtime modules:

  1. Semantic Anomaly Triggering: A rule engine inspects the live XML view hierarchy for keyword patterns (with configurable match thresholds); when a rule fires, it invokes a standalone APK that overlays a templated interruption dialog (e.g., location permission). A two-stage pipeline then executes follow-up ADB commands based on the agent's response—e.g., accepting a permission redirects to system settings, denying terminates the app—so each agent choice spawns a distinct execution path.

  2. State-Centric Success Validator: After every agent action, the framework checks the updated XML hierarchy against declarative goal schemas (e.g., content-desc == "Liked" on a specific resource-id). Task success is judged on the final UI state rather than action sequence correctness, tolerating detours and backtracking. Human review provides a final verification pass.

D-GARA-152 is the instantiated benchmark: 152 tasks across 8 apps (JD.com, Weibo, Bilibili, Amap, Ctrip, Amazon, Facebook, Google Maps), with interruptions drawn from five categories—System Network (42.8%), System Resource (28.3%), UX Disruption (10.5%), App Malfunction (9.2%), Permission Control (9.2%).

Two metrics are defined: SR (task success rate) and RSR (Robust Success Rate = success-with-interruption / success-without-interruption, restricted to tasks the agent could solve baseline).

Key Contributions

  • D-GARA framework: live anomaly injection via ADB + semantic XML-based trigger rules, supporting configurable follow-up state transitions per agent response.
  • D-GARA-152 benchmark: 152 annotated real-world Android tasks across 8 apps with embedded multi-category interruptions; open-sourced.
  • RSR metric: isolates interruption robustness from baseline task-solving ability, enabling fair cross-agent comparison.
  • Empirical finding: all SOTA GUI agents (including GUI-specialized ones) suffer >17.5% average SR drop under dynamic anomalies.
  • DataCollector tool enabling researchers to record human trajectories and define custom interruption/validation rules.

Results

  • Gemini-2.5-flash: SR 80.26% → 68.42% (RSR 73.77%) — best overall robustness.
  • GPT-4o: SR 69.08% → 60.53% (RSR 66.67%).
  • Qwen2.5-VL-7B: SR 69.08% → 46.05% (RSR 53.33%).
  • UI-TARS-1.5-72B: SR 50.66% → 39.47% (RSR 48.05%).
  • AgentCPM-GUI-8B: SR 59.87% → 26.97% (RSR 39.56%) — largest absolute drop (33pp).
  • GUI-specialized agents (UI-TARS, AgentCPM) show weaker RSR than general MLLMs, suggesting GUI training improves perception but not planning under novel events.
  • Forcing agents through the "complex" interruption path (single-button, no easy dismiss) collapses RSR dramatically: AgentCPM drops to 9.30% vs. 82.35% in dual-button mode.
  • Removing XML coordinates (screenshot-only) drops Gemini-2.5 SR by ~35pp; AgentCPM drops less, suggesting partial learned coordinate prediction.
  • "Perceptual drift" after app crashes: GPT-4o correctly relaunches the app but follows stale action history, skipping re-entry of a required search term.

Limitations

  • Benchmark size is small (152 tasks, 8 apps), limiting statistical power and generalization claims.
  • Android-only; no coverage of desktop, web, or iOS GUI agents.
  • Success validation relies on XML attribute matching—brittle to app updates that change resource-ids or content-desc values.
  • Human review of automated validation is described as a "final safeguard" but not systematically evaluated for inter-annotator agreement or error rate.
  • Interruption taxonomy is manually designed; coverage of all real-world anomaly types is not guaranteed.
  • No ablation on anomaly injection timing or frequency effects on results.

Relevance to Agentic AI / LLM Agents

D-GARA directly targets a critical gap in GUI agent evaluation: the assumption that static, idealized benchmarks predict real-world performance. The finding that GUI-specialized agents (UI-TARS, AgentCPM) are more brittle than general MLLMs under dynamic anomalies reveals that current post-training pipelines optimize grounding and action prediction without hardening planning under unexpected state transitions—an important signal for robustness-aware training research. The "perceptual drift" failure mode (agent decisions dominated by stale action history after state disruption) connects to broader challenges in agentic memory and context management for long-horizon tasks. The framework's extensible rule-based injection system provides a practical scaffold for the community to generate adversarial robustness data for GUI agents.