Skip to content

Toward AI VIS Co-Scientists: A General and End-to-End Agent Harness for Solving Complex Data Visualization Tasks

🕒 Published (v1): 2026-05-20 23:49 UTC · Source: Arxiv · link

Why this paper was selected

End-to-end agent harness for scientific visualization; concrete harness architecture case study

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a multi-agent harness called "VIS co-scientist" that autonomously designs end-to-end scientific visualization applications (VIS Apps) from only a dataset and a high-level task description. The system orchestrates specialized subagents (EDA, Planner, Environment Builder, VIS Designer, Evaluator) through explicit artifact handoffs and Playwright-based browser validation. It is validated on IEEE SciVis Contests spanning 2021–2026 across diverse scientific domains.

Problem

General-purpose AI coding assistants handle well-defined, isolated subtasks but cannot bridge the gap from an ambiguous scientific question to a deployable, interactive VIS App with streaming/volumetric data, coordinated linked views, visual encoding design, and task-driven validation. No prior system addressed end-to-end autonomous visualization design as a long-horizon, multi-stage agentic problem.

Method

The harness wraps a main OpenAI Codex (GPT-5.4) Orchestrator with five specialized subagents and supporting infrastructure:

  1. Exploratory Data Analyzer (EDA): Profiles schema, scale, missing values, and task-relevant variables; outputs an EDA Report.
  2. Planner: Translates the EDA Report and task into a concrete design specification (encodings, panels, interaction state, risks, validation checks).
  3. Environment Builder: Selects and installs minimal dependencies; records execution details to localize failures.
  4. VIS Designer: Implements complex views (3D volume rendering, progressive streaming, custom brushing, linked selections) in a render-debug-verify cycle.
  5. Evaluator: Uses Playwright-MCP to load the live app in a browser, exercise controls, check linked-view synchronization, capture screenshots, inspect console errors, and score the app against task goals.

Agents communicate through explicit document artifacts (EDA Report → Design/Implementation Report → Coding Environment → VIS App → Feedback/Fixes Scorecard) rather than implicit conversation history. A Memory Maintainer distills run insights into a hierarchical, human-readable Markdown wiki (inspired by Karpathy's LLM Wiki) for future retrieval. Custom skills (VIS Skill, Frontend Skill, Scratchbook Skill) and MCP connectors extend the base agent's capabilities.

Key Contributions

  • Defines the AI VIS co-scientist concept, enumerating four capability requirements: domain understanding, exploratory data analysis, interactive visualization design, and quality assessment.
  • Proposes a reusable multi-agent harness with explicit artifact contracts, closed-loop browser validation, and layered evaluation separating mechanistic correctness from task-level scientific utility.
  • Introduces a hierarchical Markdown wiki memory layer for post-run knowledge capture and auditability across sessions.
  • Validates feasibility on six IEEE SciVis Contests (2021, 2023, 2024, 2025, 2026) covering climate science, materials discovery, sonar imaging, neuroscience, and mantle convection.
  • Conducts a structured expert audit (5 visualization researchers, 1–5 Likert scale) on the 2025 materials-discovery case.

Results

  • 2025 materials-discovery case: Produced a five-panel linked VIS App (candidate landscape, PCA embedding, relationship matrix, pathway panel, ranking/compare view) using 368,616 input tokens + 5,183,488 cached tokens + 58,374 output tokens; runtime ~2 hours; Evaluator marked all criteria as passing.
  • Expert audit scores (mean, 1–5): EDA assumptions 4.6, EDA implementation insights 4.6, design principles 4.0, evaluation guidance 4.4, visual encodings 3.8, interface responsiveness 4.0, coordinated views 4.6, task completion 4.2; unanimous agreement (5.0) on removing all-NaN fields, strength-performance tradeoff, and hardness–yield-strength correlation; weakest agreement on 6082[%] influence statement (mean 2.4).
  • Cross-contest generalization: All runs on 2021, 2023, 2024, and 2026 contests produced mechanically valid VIS Apps from a single high-level instruction; the harness was not tuned to the materials case.
  • No numerical comparison against a single unified baseline; qualitative comparison shows a baseline coding agent (same prompt, no harness) produced disconnected plots that failed to follow key instructions.

Limitations

  • Does not yet exhibit creative visual design beyond established patterns; contest-winning entries typically require novel visual encoding contributions.
  • Multi-step visual perception and temporal reasoning are weak: Playwright screenshots and DOM checks cannot reliably validate intermittent sequences of camera movement, filtering, and linked state changes in 3D views.
  • The memory wiki is not yet active at inference time for the reported results; memory retrieval impact on performance is unmeasured and deferred to future work.
  • Token cost is high (~5.6M tokens per run); memory augmentation is hypothesized but unvalidated as a cost-reduction strategy.
  • Uncertainty quantification and trustworthiness assessment over long task durations remain open problems.
  • Expert audit participants are co-authors, limiting independence.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct, worked example of a domain-specific agent harness — a structured execution environment that wraps an LLM with subagent roles, artifact-passing contracts, tool access policies, and closed-loop evaluation — and is framed explicitly in those terms. It concretizes several harness design patterns directly relevant to this topic: artifact-as-contract between stages (preventing implicit state leakage), layered validation separating mechanistic from semantic correctness, and a wiki-based persistent memory layer for cross-session knowledge accumulation. The explicit acknowledgment that the system "aligns with Anthropic's recommendation on long-running agent design" and the structured role decomposition (Orchestrator, specialist subagents, evaluator-in-loop) make it a case study in translating harness-design principles into a real scientific domain.