Skip to content

AnovaX: A Local, Multi-Agent Voice Assistant with LLM Planning, Typed Executors, and Adaptive Recovery

🕒 Published (v1): 2026-07-16 18:09 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AnovaX is a local, ~1,800-line Python desktop voice assistant that separates LLM planning (single Gemini call emitting a JSON plan) from execution (a typed-child-agent orchestrator on a bounded thread pool). When the static plan hits a core-step failure, a bounded ReAct recovery loop takes over, hiding Gemini latency behind speculative pre-execution of read-only tools. The paper is a system description with qualitative evaluation; no quantitative benchmarks are reported.

Problem

Cloud voice assistants (Siri, Alexa) send audio off-device, expose a fixed skill set, and give users no visibility into LLM reasoning or tool calls. Prior local assistants used straight-line executors that block on any single-step failure and cannot parallelize independent actions. AnovaX targets three simultaneous properties: on-device audio privacy, real desktop manipulation (open/type/hotkey), and every tool call inspectable and whitelisted before execution.

Method

Static path. A single LLM call (Gemini) returns a JSON plan {steps, final_response, remember}. A two-stage safety filter (prompt-level rules + Python whitelist/denylist, ≤8 steps, ≤2 nesting levels, ≤20 total child agents) gates every plan. Passing plans are handed to Orchestrator.dispatch(), which groups parallel-safe steps into concurrent batches and spawns one typed ChildAgent subclass per step on a thread pool capped at 8 concurrent workers. Each subclass (AppAgent, TypingAgent, BrowserAgent, MediaAgent, InfoAgent, TimingAgent, DialogAgent, MetaAgent) carries its own TTL, retry budget, and named resource locks (kb_mouse, tts). A janitor thread running every 500 ms enforces TTLs. MetaAgent.do_work() re-invokes gemini_plan() on a sub-goal, re-runs safety_check(), and recursively re-enters the same orchestrator, hard-capped at depth 2.

Adaptive recovery path. When dispatch() returns with a failed core step, AutonomousLoop.run() activates. It issues a compact batched ReAct prompt to Gemini requesting {thought, actions, done, summary}. Simultaneously, a speculator scans the remaining steps from the failed static plan, pre-executes any read-only tools (get_time, get_date, screenshot) into a (tool, params)-keyed cache. When Gemini responds, cache hits are served directly, hiding the ~1s planning round-trip. The loop terminates on done: true, after 6 turns, or after 20 cumulative child agents. Every recovery batch passes the same safety_check().

Memory. Three layers: rolling 20-turn RAM history, 5-minute session-state (most recently opened app), and persistent memory.json facts written via the plan's remember field.

Key Contributions

  • Typed child-agent orchestrator: 10 tools map 1:1 to ChildAgent subclasses with per-class TTL, retry policy, and resource locks; concurrency capped at 8 workers.
  • Two-stage safety filter (prompt rules + Python whitelist/denylist) applied to every plan including recursively generated sub-plans, before any agent is spawned.
  • Adaptive recovery via bounded batched ReAct (6-turn, 20-agent budget) with speculative parallelism: read-only tools pre-run against a cache during the Gemini planning round-trip.
  • MetaAgent recursive delegation: sub-goals re-enter the full plan→safety→dispatch pipeline, capped at depth 2.
  • Three-layer memory and an SSE lifecycle event bus mirrored to desktop UI, JSONL log, and phone remote; MJPEG screen-stream back to phone.

Results

Evaluation is entirely qualitative (hand-checked, 5 runs each):

  • 13 request categories verified: app launch, launch-and-type, browser open, type-into-window, web search, YouTube search, time/date, screenshot, small talk, parallel read-only batch, recursive sub-plan, multi-step compound, refusal cases — all pass on ≥4/5 runs.
  • Latency ranges (non-benchmark, session log): speech recognition 0.6–1.4 s; Gemini plan 0.7–2.0 s; safety check + dispatch <0.05 s; lock acquisition <0.01 s; TTS render 1.2–2.0 s.
  • Ablation — Gemini disabled: regex fallback handles 8/13 categories; fails on recursive sub-plan, paraphrased launches, parallel batching, novel jokes.
  • Ablation — locks disabled: ~1 in 5 multi-step runs exhibits a press_keys/type_text race.
  • Ablation — autoloop disabled: wrong-launcher failure surfaces as a hard error; with autoloop, Gemini retries via web_search in one additional round-trip; speculative cache cuts perceived recovery latency by ~0.7 s on the tested example.
  • Refusals fire correctly in all ablation modes.

Limitations

  • Blind executor: typed agents fire pyautogui events without verifying the resulting screen state; a step can "succeed" (no exception) into the wrong window.
  • pyautogui fragility on Windows: the kb_mouse lock removes intra-plan races but not OS-level focus interruptions (e.g., Windows Defender scan stealing focus after app launch).
  • No quantitative benchmark: all evaluation is hand-checked on a single machine; generalization claims are unsupported by data.
  • Cloud dependency: Gemini API calls leave the machine; truly offline operation falls back to a regex parser covering only a subset of capabilities.
  • Fixed schema brittleness: adding a new tool requires edits in three places (system prompt, whitelist, agents.py); the executor cannot observe whether intended effects occurred.
  • Scope: 1,800-line proof-of-concept; not evaluated for robustness, accessibility, or multi-user scenarios.

Relevance to Harnesses / Meta-Harnesses

AnovaX is a concrete, readable implementation of a two-tier harness: an outer orchestrator that manages agent lifecycle, concurrency, resource locks, TTLs, and safety gating, and an inner recovery harness (the AutonomousLoop) that switches execution strategy when the primary plan fails. The MetaAgent pattern — where a child agent re-enters the full plan→safety→dispatch pipeline — is a micro-meta-harness, showing how recursive self-delegation can be bounded and safety-wrapped at each level. The speculative parallelism technique (pre-running read-only tools into a cache during an LLM planning call) is a latency-hiding pattern directly applicable to any harness that interleaves model inference with tool execution. The explicit separation of "orchestrator decides dispatch" from "LLM decides plan" is the defining architectural choice that makes the harness auditable and the safety layer enforceable.