Skip to content

Sensorium Arc: AI Agent System for Oceanic Data Exploration and Interactive Eco-Art

🕒 Published (v1): 2025-11-20 02:48 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Sensorium Arc is a real-time multimodal AI agent system deployed as an interactive art installation that embodies the ocean as a poetic conversational persona, grounding responses in scientific marine datasets via RAG. It uses a modular multi-agent pipeline—visualization selector, query rewriter, and responder—running locally in Unity to translate user whispers into voice, globe visualization, and audiovisual output. The system demonstrates conversational AI as a medium for affective public engagement with high-dimensional climate data.

Problem

Ocean environmental datasets (CO2, chlorophyll, sea surface temperature, hyperspectral Kd) are publicly available but inaccessible to general audiences due to the absence of intuitive, emotionally resonant interfaces. Standard static visualizations fail to generate the affective engagement needed for ecological literacy and behavioral change.

Method

A Unity-based multi-agent RAG system with four layered components:

  1. Input Processing: Arduino-controlled Nautilus shell interface with a proximity sensor (50 cm threshold), LED, microphone with active noise cancellation, and Whisper-tiny for speech-to-text.
  2. LLM Pipeline (three stateless agents):
  3. Visualization Decider Agent (Llama-3.2-3B): selects a data visualization token via few-shot prompting + GBNF grammar constraints before response generation.
  4. Retrieval & Query Rewriter Agent (Qwen 8B + chain-of-thought): rewrites user queries to align with the corpus, then retrieves k=2 sentence/paragraph pairs from a 384-dim vector index (all-MiniLM-L12-v2, ANN via usearch) over a curated eco-art corpus (Harrison archive essays, manifestos, catalogs).
  5. Responder Agent (Llama-3.2-3B or Gemma 3 12B): generates the Ocean persona response from visual description, retrieved passages, and user query.
  6. Response Processing: keyword matcher maps output tokens to audiovisual triggers; Jets TTS converts text to audio with subtitles.
  7. Audio-visual Layers: NASA EarthData globe overlays (CO2, chlorophyll, SST, ocean currents, Kd from PACE satellite), pre-rendered Unity videos (plankton blooms, ocean acidification, plastic dispersion, sea-level rise).

Average response latency: <4 seconds on RTX 4090 Laptop GPU with CUDA layer offloading.

Key Contributions

  • Multi-agent RAG architecture decomposing visualization selection, query rewriting, and persona-grounded response into specialized stateless agents to reduce prompt interference.
  • Upstream visual selection token mechanism ensuring the Responder Agent is aware of the chosen visualization before generating its narrative.
  • Curated eco-art RAG corpus (Harrison archive) enabling poetic grounding without domain-specific fine-tuning, with the corpus updatable without model retraining.
  • Embodied physical interface (Nautilus shell, proximity trigger, TTS with subtitles) enabling public exhibition in mobile form factor.
  • Integration of hyperspectral Kd data from NASA's PACE satellite (2024) rendered as 16 stacked globe layers.

Results

  • No formal quantitative benchmark is reported; the paper is a system/design paper presented at NeurIPS 2025 Creative AI Track.
  • Exhibition deployment demonstrated real-time voice-to-voice interaction with <4 second end-to-end latency.
  • Comparative agent model selection: Qwen 8B outperformed other candidates on rule-following for query rewriting; Gemma 3 12B selected over Llama 3.2 3B for the Responder when quality required larger context.
  • k=2 retrieval chosen empirically—larger k degraded response coherence by filling the Responder's context window.

Limitations

  • No controlled user studies; claims about empathy, retention, and behavioral impact are aspirational, not measured.
  • RAG reliability is limited: k=2 retrieval is low-diversity; no query expansion or document-type categorization currently implemented.
  • Video layers are pre-rendered (not real-time), limiting interactivity with visualizations.
  • GPU contention between Unity rendering and LLM inference causes latency pressure; cloud offload planned but not implemented.
  • Corpus is English-only and restricted to the Harrison archive, limiting scientific and cultural diversity.
  • Multi-agent coordination issues (role definition, inter-agent alignment, output verification) are acknowledged as open challenges per cited literature.

Relevance to Agentic AI / LLM Agents

This work is a concrete example of a deployed multi-agent system where task decomposition into specialized, stateless agents (decider, rewriter, responder) demonstrably reduces prompt interference compared to a monolithic LLM—a practical validation of agent specialization principles discussed in the broader multi-agent LLM literature. The upstream visual selection token pattern (deciding grounding context before generation) is a reusable architectural motif for any agent system requiring multimodal coherence between retrieval and output. The use of RAG with a domain-specific corpus for persona grounding—without fine-tuning—illustrates a generalizable pattern for lightweight domain adaptation in agentic pipelines. For researchers tracking agentic AI, this is a rare deployed, exhibition-tested system that surfaces real-world failure modes (context window pressure, inter-agent alignment, GPU/LLM resource contention) beyond benchmark settings.