Just A Rather Very Intelligent Spoken Agent¶
🕒 Published (v1): 2026-07-18 03:10 UTC · Source: Arxiv · link
Why this paper was selected
Voice-interactive long-horizon agent with richer user feedback channels beyond text; modality extension for harnesses
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
The paper introduces JarvisBench, a plug-and-play benchmark for evaluating a spoken mediator layer ("Jarvis") that sits between users and long-horizon worker agents. Jarvis monitors the worker's ReAct execution trace, supports always-on spoken user interaction, and injects sparse user guidance at failure-sensitive checkpoints. Preliminary results on 34 WildClaw tasks show task-completion gains up to +11.78% (absolute) over unmediated baselines.
Problem¶
Long-horizon AI agents operate in a thin, turn-based interaction paradigm: users issue an initial instruction and receive only sparse textual updates, leaving them unable to monitor progress, correct early errors, or steer execution before the agent drifts. No standardized benchmark exists to measure whether a communication mediator between user and agent improves both task outcomes and real-time user awareness.
Method¶
JarvisBench defines a two-track evaluation protocol for a mediator placed between a user and an unmodified worker agent:
Track 1 (Agent-Collaboration): The same worker agent runs with and without Jarvis. Jarvis monitors the worker's ReAct trajectory and triggers sparse, failure-sensitive checkpoints (repeated tool failures, stalled progress, ambiguous observations, risky irreversible actions, low-confidence finalization). At each checkpoint, Jarvis summarizes the relevant execution state, solicits concise guidance from a GPT-backed expert-user simulator, and injects that feedback back into the worker's ongoing context. The worker remains the primary task executor; Jarvis cannot solve sub-tasks independently.
Track 2 (User-Interaction): The worker agent is fixed; the Jarvis brain is swapped. A nonexpert-user simulator asks up to four short natural-language questions per task (alternating progress-oriented and topic-oriented) at scheduled checkpoints. Jarvis answers through a cascaded full-duplex speech pipeline (streaming ASR → semantic VAD via SoulX-Duplug → TTS via Chatterbox). Answers are judged by an LLM against the execution trace (progress) or original task context (topic). Latency is measured end-to-end from user utterance end to Jarvis speech start.
The prototype separates dialogue context (recent user-Jarvis turns, user preferences) from execution context (worker actions, observations, errors, current plan) to prevent conflation of conversational and monitoring demands.
The benchmark is plug-and-play: any component—Jarvis brain, speech stack, monitoring policy, memory design, worker agent—can be swapped independently so long as the system exposes required interaction logs.
Key Contributions¶
- JarvisBench benchmark with two complementary tracks (agent-collaboration, user-interaction) targeting the underexplored mediator role in long-horizon agent workflows.
- Modular reference Jarvis prototype with a cascaded full-duplex speech interface, ReAct trace monitoring, separated context management, and cooperative interaction checkpointing.
- Plug-and-play evaluation protocol enabling independent substitution of LLM brain, speech stack, monitoring/checkpointing policy, or worker agent under a fixed two-axis metric framework.
- Empirical evidence that sparse, user-grounded guidance injection improves task completion without replacing the worker agent, and that mediator brain capability is the dominant factor in both tracks.
Results¶
Agent-Collaboration Track (34 WildClaw text-only tasks, GPT-5.4 as Jarvis brain):
| Worker Agent | Baseline | +Jarvis | Δ | # Full-score gain |
|---|---|---|---|---|
| GPT-5.4-mini | 44.24% | 49.11% | +4.86% | 4→6 |
| GPT-5.5 | 50.54% | 58.45% | +7.90% | 6→8 |
| Gemini-3.1-Pro | 43.14% | 54.78% | +11.64% | 6→8 |
| Claude Opus 4.7 | 64.01% | 75.79% | +11.78% | 10→14 |
User-Interaction Track (fixed GPT-5.4-mini worker; varying Jarvis brain):
- GPT-5.4 achieves the best topic score (3.91 avg; 73.9% ≥4) at 1.39s latency.
- GPT-OSS-120B achieves the best progress score (4.07 avg; 93.8% ≥4) at 1.12s latency.
- Qwen3 models are fastest (0.69–2.89s) but weakest on topic grounding (41.5–52.0% ≥4).
- GPT-5.4-mini closely matches GPT-5.4 on progress (4.02; 92.7% ≥4), indicating smaller proprietary models suffice for trace-grounded status reporting.
Limitations¶
- Evaluated on only 34 text-only tasks; multimodal WildClaw tasks (image, video, audio, OCR) are excluded, limiting generality.
- User simulators are LLM-backed (GPT-based), not real humans; simulated expert feedback may not reflect the variance or quality of real-domain experts.
- Intervention checkpointing introduces a cooperative pause mechanism that may not reflect truly asynchronous real-world deployment.
- Results are described as "preliminary"; no statistical significance or confidence intervals are reported.
- Mediation gains depend strongly on the Jarvis brain's capability; weaker LLMs introduce poorly timed interventions, limiting applicability with open models.
- The benchmark covers only text-based ReAct worker loops (OpenClaw/Hermes); applicability to other execution paradigms (DAG planners, code interpreters, multi-agent graphs) is untested.
Relevance to Harnesses / Meta-Harnesses¶
JarvisBench is itself a meta-harness: a plug-and-play evaluation wrapper that surrounds an existing agent execution loop (OpenClaw) and instruments it with a mediation layer, interaction checkpoints, and a two-track logging/judging protocol. The plug-and-play design—where any single component (LLM brain, speech stack, monitoring policy, worker agent) can be swapped while the evaluation protocol remains fixed—is a canonical meta-harness pattern, directly analogous to how frameworks like OpenDevin or SWE-bench wrap diverse agent implementations under a unified evaluation surface. The paper's checkpointing mechanism (cooperative pauses that expose worker state, collect user guidance, and resume execution) is a concrete implementation of harness-level intervention hooks, a technique directly relevant to anyone building or evaluating meta-harnesses that must orchestrate human-in-the-loop feedback into running agent pipelines. The distinction between the mediator's role (communication bridge, not solver) and the worker agent's role also illustrates a clean architectural separation between harness-level control logic and task-execution logic.