Skip to content

HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization

🕒 Published (v1): 2026-06-25 05:19 UTC · Source: Arxiv · link

Why this paper was selected

HITL agentic system design prioritizing transparency over autonomy; concrete design lessons for human-in-loop harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HiLSVA is a human-in-the-loop agentic system for scientific visualization (SciVis) that combines a plan-first multi-agent architecture with explicit human oversight, stepwise provenance tracking, and learn-at-test-time (LTT) adaptation from user feedback. It addresses the gap between fully autonomous SciVis agents and the inherently human-centered nature of scientific analysis. A 12-participant user study confirms that mixed-initiative interaction improves task completion and transparency at the cost of execution efficiency.

Problem

Existing LLM-based SciVis agents (e.g., ChatVis, VizGenie, NLI4VolVis) prioritize autonomy and task completion over human analytical control, sacrificing transparency, oversight, and accountability. No prior system simultaneously offers SciVis domain adaptation, mixed-initiative interaction, workflow provenance, sandboxed safe execution, and test-time adaptation from human feedback.

Method

HiLSVA is built on top of Magentic-UI and extends it with SciVis-specific capabilities:

  • Plan-first orchestration: A lead orchestrator synthesizes an explicit stepwise plan, assigns each step to a specialized sub-agent, and exposes the plan as an editable artifact for user review/approval before execution begins.
  • Specialized agents: A ParaView agent (via ParaView-MCP for parameterized API calls), a code agent (RAG-enhanced, iterative error-correcting Python/shell script generator inspired by ChatVis), a web surfer, a file surfer, and a self-improving agent.
  • Mixed-initiative execution: Users can intervene at any step via natural language or direct GUI manipulation (e.g., adjusting transfer functions); control fluidly transfers back to agents afterward. Autonomy level is tunable.
  • Sandboxed execution: ParaView runs inside isolated Docker containers; critical/irreversible actions require explicit user approval (action guards).
  • Stepwise provenance: Every action, software state, and visualization output is recorded, supporting undo, branching, and workflow reuse.
  • Learn-at-test-time (LTT): A self-improving agent reflects on execution outcomes, detects uncertainty, proactively queries the user, and writes feedback into a dynamic knowledge repository that persists across tasks within a session.

Key Contributions

  • Mixed-initiative SciVis architecture combining human oversight, co-planning, co-execution, and autonomy-level control — the only system in their comparison table to satisfy all six capability dimensions simultaneously.
  • Stepwise provenance tracking enabling undo, alternative trajectory comparison, and workflow reuse in SciVis contexts.
  • LTT adaptation: the self-improving agent updates a dynamic knowledge base from live human feedback during inference, without retraining.
  • Backend-agnostic design: SciVis engines (VMD, napari, TTK) can be swapped in by providing an MCP server or equivalent tool interface.
  • Controlled user study (N=12, three autonomy modes, mixed expertise) evaluating the autonomy–oversight tradeoff in agentic SciVis.

Results

  • All twelve participants across expertise levels (novice to domain scientist to SciVis expert) successfully completed non-trivial SciVis tasks with HiLSVA support.
  • Mixed-initiative interaction improved task completion, user control, and workflow transparency relative to fully autonomous mode across all expertise groups.
  • Higher automation yielded better execution efficiency; increased human involvement yielded better control and oversight — a clear tradeoff.
  • Post-study feedback indicated strong perceived usability, transparency, and support for iterative refinement.
  • No quantitative accuracy/speed numbers are reported in the provided text beyond the Copilot reference (>50% faster, cited from prior work [51], not HiLSVA itself).

Limitations

  • Evaluation relies on 12 participants; no large-scale reproducible benchmark is provided, by the authors' own admission (human-in-the-loop studies resist benchmark-style scaling).
  • Current implementation is ParaView-specific; backend-agnostic claims are architectural, not empirically validated for other engines.
  • LTT knowledge base is session-scoped; long-term adaptation across sessions is not addressed.
  • No ablation isolating the contribution of individual components (provenance, LTT, action guards) to task outcomes.
  • The tradeoff between efficiency and oversight is identified but not quantified (e.g., no wall-clock or step-count numbers in the provided text).

Relevance to Agentic AI / LLM Agents

HiLSVA is a concrete instantiation of the broader push toward human-in-the-loop agentic systems, directly complementing frameworks like Magentic-UI and HULA by grounding mixed-initiative design in a demanding domain (SciVis) where full autonomy is both epistemically risky and undesirable. The plan-first, editable-artifact workflow and the LTT self-improving agent are transferable patterns for any domain where correctness depends on human judgment that cannot be fully specified upfront. The empirical finding that mixed-initiative interaction outperforms full autonomy on transparency and task completion — while trading off efficiency — provides evidence directly relevant to the human-oversight vs. automation debate central to agentic AI research. The architecture's use of MCP for domain-specific tool integration is also a concrete example of the emerging MCP-as-agent-infrastructure pattern.