Skip to content

Voice-Interactive Surgical Agent for Multimodal Patient Data Control

🕒 Published (v1): 2025-11-10 18:47 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VISA is a hierarchical multi-agent framework for robotic surgery that interprets free-form voice commands to control multimodal patient data (clinical records, CT scans, 3D anatomical models) overlaid on live surgical video. An LLM-driven orchestration agent coordinates three task-specific agents (IR, IV, AR) through a hybrid probabilistic-rule-based workflow. The system introduces MOEM, a two-level evaluation metric assessing both per-stage accuracy and end-to-end workflow success.

Problem

Robotic surgeons cannot divert hands or eyes to access patient data during procedures. Prior voice-control systems relied on predefined command sets or single-LLM direct mapping, which fail on free-form natural language, medical-term ASR errors, implicit/context-dependent commands, and multi-step composite instructions.

Method

VISA is a hierarchical MAS with one orchestration agent and three task-specific agents, all backed by LLMs (Gemma3 for reasoning, Whisper for STT):

  • Orchestration agent: at each step, the LLM outputs a JSON of {function_name: probability} over all workflow functions; a deterministic function-selection rule picks \(f_{\text{next}} = \arg\max_{f} p(f)\); missing functions are filled with \(p=0\) (missing-function-completion rule). This hybrid LLM + rule approach prevents mispredictions at branching points.
  • Workflow functions: real-time audio (Silero-VAD + Whisper-small + TTS), STT (Whisper-medium, beam=8), correct-and-validate (Gemma3 with CoT, medical-term correction rules, global memory context), command reasoning (Gemma3 selects target agent).
  • Task agents (IR, IV, AR): each has an action-determination (AD) function that jointly infers action \(a^\star\) and state parameters \(\Psi^\star\) via an LLM policy \(\pi_\theta(a, \Psi \mid x, i)\). Agents update local clip memory and a shared global memory of the last three commands and agents, enabling resolution of implicit and ambiguous commands.
  • Memory: local memory per 10-second clip; global memory shared across clips for cross-context disambiguation.

Key Contributions

  • First hierarchical multi-agent framework for hands-free surgical data interaction, with an orchestrator + three domain-specific agents operating on live robotic-surgery video.
  • Hybrid orchestration strategy combining LLM probabilistic output with deterministic selection and fallback rules for reliability.
  • MOEM (Multi-level Orchestration Evaluation Metric): quantifies stage-level accuracy (STT, CC, CR, AF, AP, OF) and workflow-level success rate, plus command-category breakdowns (structure, type, expression).
  • Curated 240-command dataset annotated along three hierarchical dimensions (structure: single/composite; type: explicit/implicit/NLQ; expression: baseline/abbreviation/paraphrase).
  • Medical-term correction pipeline (STT post-processing) and contextual disambiguation via rolling global memory.

Results

  • Stage-level accuracy reported across STT, CC, CR, AF, AP, and OF stages; VISA achieves "high stage-level accuracy and workflow-level success rates" on the 240-command dataset (exact per-stage numbers are in Table II, which is truncated in the provided text).
  • Robustness validated against three linguistic variation categories: transcription errors (abbreviations, medical homophones), linguistic ambiguity (implicit/NLQ commands), and free-form paraphrases.
  • Synthesized speech from four TTS voices tested alongside real speech to assess generalization.
  • Composite commands (15 instances) and implicit/NLQ commands (80 each) included; the MOEM category breakdown identifies which command types are most error-prone.

(Note: precise accuracy/SR percentages are not reproduced in the provided text excerpt.)

Limitations

  • Dataset is small (240 commands) and sourced from a single patient's surgical case, limiting generalizability.
  • Evaluation restricted to synthesized and one real-patient scenario; no multi-patient or multi-surgeon study.
  • 10-second clip segmentation constrains single-command-per-clip design; complex multi-step interleaving across clips is not demonstrated.
  • Latency analysis is qualitative; no hard real-time latency guarantees reported.
  • Gemma3 dependency for all reasoning functions; no ablation against other LLM backends reported in the excerpt.

Relevance to Harnesses / Meta-Harnesses

VISA is a concrete instantiation of a domain-specific meta-harness: a top-level orchestrator that dynamically routes to specialized sub-agents, mirroring the orchestrator-worker pattern central to agent harness design. The hybrid dispatch mechanism—LLM outputs a probability distribution over callable functions, then a deterministic rule selects the next function—is a practical recipe for reliable harness control flow where pure LLM decision-making is fragile. The rolling global-memory state that persists agent identity and parameters across invocations is directly analogous to harness-level context management across tool calls. MOEM's decomposition of orchestration quality into per-stage and end-to-end metrics offers a reusable evaluation template for any multi-step agentic harness.