Skip to content

KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

🕒 Published (v1): 2026-07-15 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Self-evolving GUI agent with memory and skill accumulation; cross-platform deployment relevance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

KnowAct-GUIClaw is a personal GUI assistant framework that extends OpenClaw-style agent runtimes with cross-platform GUI interaction and a self-evolving memory/skill system. It follows a Know–Route–Act–Reflect loop where accumulated trajectory experience is distilled into retrievable memory and validated, parameterized skills that improve future task execution without model retraining. On the MobileWorld long-horizon benchmark, the system with open-source Kimi-2.6 achieves 64.1% success rate, outperforming closed-source models including GPT-5.5.

Problem

Existing personal assistant frameworks (OpenClaw, Nanobot, Hermes) lack (1) native cross-platform GUI interaction support—real-world tasks frequently require visual grounding across dynamically rendered interfaces where no clean API exists—and (2) any self-improvement mechanism: successful and failed trajectories are discarded after each run, forcing the agent to relearn the same shortcuts, navigation patterns, and failure modes from scratch. Integrating a standalone GUI agent naively into such frameworks introduces hallucinated app-allocation, information loss in cross-app data transfer, redundant navigation overhead, and no reuse of operational shortcuts.

Method

KnowAct-GUIClaw structures execution as a four-stage loop over two persistent stores (memory/history and skill/shortcut):

  • Know: Before acting, the host agent retrieves task-relevant app candidates, policies, and GUI hints via semantic similarity from the memory store. Policy memory is injected directly; experience memory is advisory. The host resolves subtasks itself when conversation context or non-GUI tools suffice, delegating to the GUI subagent only when live visual interaction is required.
  • Route: The router classifies a request as single-app or multi-app. For multi-app workflows, each subtask is a typed tuple \((g_i, h_i, I_i, O_i)\)—goal, optional app hint, declared inputs, declared outputs. A blackboard \(B_i\) enforces typed data flow: \(G(g_i, h_i, B_{i-1}[I_i]) \to \tau_i,\; E(\tau_i, O_i) \to B_i[O_i]\). Missing inputs or outputs cause fail-closed behavior rather than inference or fabrication.
  • Act: GUIClaw runs an observe–reason–act loop over a hybrid action space \(A = A_\text{gui} \cup A_\text{skill} \cup A_\text{shortcut} \cup A_\text{ask}\). Skills are retrieved (top-5) by lexical+embedding signals and validated via deterministic state contracts or visual valid-state checks before use. Android deeplinks/intents in \(A_\text{shortcut}\) are used only after target-page and parameter validation.
  • Reflect: Post-run, trajectories are summarized and selectively distilled into parameterized skills (vision LLM rewrites GUI events into structured procedures with state contracts) or experience memory. Skill evolution repairs existing skills in-place when they fail, taking precedence over new extraction.

Key Contributions

  • Two-tier host–executor separation: host handles orchestration and long-horizon context; lightweight GUI subagent handles low-level visual interaction, with structured resumable task interfaces and partial-completion checkpoints.
  • Blackboard-mediated information contracts: typed input/output declarations per subtask enforce cross-app data integrity and prevent value hallucination.
  • Experience-attributable memory system: retrieves app-specific GUI hints, lessons, and routing candidates to guide decomposition before any action is taken.
  • Self-evolving skill library: skills are extracted from trajectories as parameterized, state-validated procedures; failed skill reuse triggers in-place repair rather than replacement; skills and memory transfer across base models.
  • Hybrid action space with validated shortcuts: Android deeplinks and intents bypass GUI navigation only after state contract verification, avoiding unsafe broad-intent execution.

Results

  • MobileWorld (GUI-Only, long-horizon): KnowAct-GUIClaw with Kimi-2.6 achieves 64.1% success rate (SR), outperforming all agentic frameworks and closed-source models including Seed-2.0-Pro and GPT-5.5.
  • Memory+skill transfer: Adding the framework's memory and skills to Kimi-2.6 improves SR by +8.5%; for Qwen3.5-35B-A3B the improvement is +16.2%, demonstrating model-agnostic transferability.
  • AndroidDaily: Results reported (exact numbers not provided in the supplied excerpt).
  • Cross-platform validation: System deployed and evaluated on Android, iOS, HarmonyOS, and Windows.
  • Ablation confirms both memory and skill components contribute independently to performance gains (Figure 1).

Limitations

  • Skill extraction and evolution rely on a vision LLM post-run; quality of distilled skills is bounded by that model's ability to infer correct state contracts from trajectory screenshots.
  • Skill validation via state contracts may fail silently on rare or unseen UI states, degrading to ordinary GUI execution rather than causing an error.
  • The blackboard fail-closed policy means any missing intermediate value aborts the entire multi-app workflow—no graceful partial recovery.
  • Shortcut integration is currently Android-specific (deeplinks, intents); generalization to iOS, HarmonyOS, and Windows shortcut mechanisms is not detailed.
  • Benchmark results are on MobileWorld and AndroidDaily; coverage of web/desktop benchmarks (OSWorld, WebArena) is absent from the presented results.
  • The paper text is truncated; ablation numbers and AndroidDaily figures are not fully available for review.

Relevance to Agentic AI / LLM Agents

This paper directly addresses a core open problem in agentic systems: how to make agents improve through deployment experience without retraining. The Know–Route–Act–Reflect loop is a concrete instantiation of the write–manage–read memory paradigm applied to GUI-facing agents, with validated executable skills serving as a durable behavioral layer above raw LLM inference. The typed blackboard contract for cross-agent data flow is a clean engineering answer to the information-loss problem in multi-agent pipelines, relevant beyond GUI contexts. For researchers tracking LLM agents, the demonstrated model-agnostic skill transfer (+8.5–16.2% across different base models) is the most practically significant result, suggesting that operational knowledge can be decoupled from model weights.