Skip to content

KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

🕒 Published (v1): 2026-07-14 11:04 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

KnowAct-GUIClaw extends OpenClaw-style agent harnesses with a two-tier host–executor architecture that adds cross-platform GUI interaction, a retrieval-augmented memory system, and a self-evolving skill library. The system follows a Know–Route–Act–Reflect loop that converts execution trajectories into reusable skills and experience memory, enabling continuous improvement without weight updates. On the long-horizon MobileWorld benchmark it reaches 64.1% success rate with open-source Kimi-2.6, outperforming GPT-5.5 and Seed-2.0-Pro.

Problem

OpenClaw-style agent harnesses lack two critical capabilities: (1) they have no native mechanism for GUI interaction—tasks requiring visual grounding, sequential tap/swipe decisions, or cross-app data transfer cannot be delegated to a clean API—and (2) they provide no self-evolution loop, so every run discards trajectory evidence and forces re-exploration of known workflows, navigation paths, and failure patterns.

Method

The framework separates concerns into a host agent (orchestration, memory, user profile, tool routing) and a GUIClaw subagent (screenshot perception, action normalization, device backends). Execution proceeds in four stages:

  • Know: Retrieves task-relevant app candidates, skills, and policy memory via semantic similarity before any action. Policy memory is injected directly; experience memory is advisory. The host resolves information-gathering subtasks itself when no live GUI state is needed.
  • Route: Classifies the request as single-app or cross-app. For multi-app workflows, each subtask is a typed tuple \((g_i, h_i, I_i, O_i)\) (goal, app hint, inputs, outputs). A shared blackboard \(B_i\) propagates values between subtasks per \(G(g_i, h_i, B_{i-1}[I_i]) \to \tau_i\), \(E(\tau_i, O_i) \to B_i[O_i]\); missing values cause hard failure rather than hallucination.
  • Act: GUIClaw operates over a hybrid action space \(\mathcal{A} = \mathcal{A}_{\text{gui}} \cup \mathcal{A}_{\text{skill}} \cup \mathcal{A}_{\text{shortcut}} \cup \mathcal{A}_{\text{ask}}\). Skills are parameterized procedures with per-step state contracts validated against live screen state before commitment. Android deeplinks/intents in \(\mathcal{A}_{\text{shortcut}}\) are used only after target-page and parameter validation.
  • Reflect: Post-run, a vision LLM extracts declarative skills from trajectory evidence under a restricted prompt. When a reused skill fails, an evolution prompt repairs it in-place (narrowing scope, adding obstacle handling, refreshing state contracts) rather than discarding it. Only runs with genuine signal are written to long-term stores.

Key Contributions

  • Two-tier host–executor interface with structured, resumable task handoffs (completed / partial / blocked) and adaptive host involvement that bypasses the GUI subagent for pure information-retrieval subtasks.
  • Typed blackboard routing that enforces explicit input/output contracts across cross-app workflows, eliminating free-form value fabrication between subtasks.
  • Hybrid action space unifying GUI primitives, state-validated skills, and deeplink/intent shortcuts, with per-step state-contract checking before skill commitment.
  • Attribution-policy memory that distinguishes injected policy memory from advisory experience memory, preventing stale traces from overriding current instructions.
  • Skill evolution via targeted repair: failed skill steps trigger in-place refinement rather than new extraction, preserving accumulated reliability counters and state contracts.

Results

  • MobileWorld (GUI-Only): KnowAct-GUIClaw with Kimi-2.6 achieves 64.1% success rate, beating all agentic frameworks and closed-source models including Seed-2.0-Pro and GPT-5.5.
  • Memory + skill transfer: Adding memory and skills improves Kimi-2.6 by +8.5% and Qwen3.5-35B-A3B by +16.2% over base-model baselines.
  • AndroidDaily: Evaluated separately; results reported in Table form (specific numbers not provided in the supplied excerpt).
  • Cross-platform: Deployed across Android, iOS, HarmonyOS, and Windows; cross-platform checks reported in §5.6 (specific numbers not in excerpt).

Limitations

  • Cross-platform performance numbers and AndroidDaily absolute scores are not fully provided in the supplied text, limiting reproduction of claimed generalization.
  • Skill extraction requires a vision LLM post-run pass, adding latency and API cost per trajectory; the extraction quality depends on the restricted prompt's coverage of edge cases.
  • State-contract validation for skills relies on visual-valid-state checks as a fallback when deterministic contracts are unavailable, which may be brittle on dynamically rendered UIs.
  • The blackboard's fail-closed semantics prevents graceful partial completion when a required output is extractable but not declared in advance.
  • Deeplink/intent shortcuts are Android-specific; the analogous shortcut mechanism for iOS, HarmonyOS, or Windows is not detailed in the excerpt.
  • Offline skill clustering is not described with enough detail to assess deduplication quality or sensitivity to structurally near-duplicate trajectories.

Relevance to Harnesses / Meta-Harnesses

KnowAct-GUIClaw is directly a meta-harness design: it wraps an existing harness (OpenClaw) with an outer loop that manages sub-agent delegation, typed data contracts, and persistent skill/memory stores that feed back into every stage of the pipeline. The Know–Route–Act–Reflect stack is a concrete instantiation of the general harness pattern where the orchestrator layer controls what sub-agents see, what they may return, and how their outputs are distilled into reusable artifacts. The blackboard-mediated information contract is a transferable pattern for any multi-agent harness that needs to prevent hallucinated cross-step data propagation. The self-evolving skill library shows how a meta-harness can implement closed-loop improvement without retraining the base model—a key capability gap in static harness frameworks like vanilla OpenClaw or LangChain-style orchestrators.