Skip to content

Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports From Scratch with Agentic Framework

🕒 Published (v1): 2025-06-03 05:18 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multimodal DeepResearcher is a four-stage agentic framework that generates text-chart interleaved research reports from scratch by orchestrating web search, in-context learning from human expert reports, structured planning, and iterative chart refinement. It introduces Formal Description of Visualization (FDV), a grammar-of-graphics-inspired structured textual representation that lets LLMs learn from and generate diverse charts. Against the adapted DataNarrative baseline, it achieves an 82% overall win rate (automatic) and 100% (human evaluation) using Claude 3.7 Sonnet.

Problem

Existing deep research frameworks (OpenAI Deep Research, OpenResearcher, Search-o1, etc.) generate text-only reports. Integrating visualizations requires: (1) a representation of charts that fits within LLM context for in-context learning, and (2) a pipeline that coherently places and styles charts relative to surrounding text. No prior work addressed end-to-end automated generation of multi-visualization reports from free-form topics.

Method

The framework decomposes report generation into four sequential stages:

  1. Researching: Iterative web search + reasoning with GPT-4o-mini. The agent generates keywords \(K = \{k_1, \ldots, k_{n_K}\}\), retrieves pages \(P = \{p_1, \ldots, p_{n_P}\}\), synthesizes learnings \(L\), and formulates the next research question, repeating for \(n_R\) rounds.

  2. Exemplar Report Textualization: Human expert multimodal reports are converted to pure-text by replacing each chart image \(i\) with its FDV: \(\tilde{r} = r.\text{replace}(i, M_v(i))\), where \(M_v\) is Claude 3.7 Sonnet as MLLM. FDV encodes four perspectives—overall layout, plotting scale, data (numeric + text), and marks (visual element specs)—inspired by Wilkinson's grammar of graphics.

  3. Planning: The LLM generates a hierarchical report outline \(O\) and a visualization style guide \(G\) (color palette, font hierarchy) from learnings \(L\), topic \(t\), and textualized exemplars \(\tilde{R}\).

  4. Multimodal Report Generation: The LLM drafts a textual report with FDV placeholders; each FDV is then implemented as D3.js code (chosen over declarative libraries for flexibility). An actor-critic loop refines each chart: a browser tool captures console errors and a screenshot; the MLLM critic \(M_v\) evaluates visual quality and returns feedback; the LLM actor \(M_t\) rewrites the code. Max retries \(N_{\max} = 3\); the critic selects the better of the last two candidates.

Key Contributions

  • Novel task definition: automated end-to-end generation of text-chart interleaved multimodal reports from a free-form topic.
  • FDV (Formal Description of Visualization): structured four-perspective textual representation enabling in-context learning and faithful chart reconstruction via code.
  • Multimodal DeepResearcher: end-to-end agentic framework with distinct stages for research, textualization, planning, and generation+refinement.
  • MultimodalReportBench: 100-topic benchmark spanning 10 categories (travel, energy, education, etc.) sourced from Pew Research, Our World in Data, and Open Knowledge Foundation, with five report-level and five chart-level evaluation metrics.

Results

  • Automatic evaluation vs. DataNarrative (Claude 3.7 Sonnet): 82% overall win rate; Verifiability 86%, Visualization Quality 80%, Visualization Consistency 78%, Informativeness 75%, Coherence 76%.
  • Human evaluation (20 topics, 5 annotators, Claude 3.7 Sonnet): 100% overall win rate; 80% agreement with GPT-4.1 automatic scores.
  • Open-source models (Qwen3-235B-A22B + Qwen2.5-VL-72B): 55% overall win rate vs. DataNarrative, with largest margin on Verifiability (66% vs. 21% lose).
  • Chart quality scores (Claude config, 1–10 scale): Readability 8.97 vs. 8.52, Layout 9.23 vs. 8.48, Aesthetics 9.12 vs. 8.38, Data Faithfulness 9.83 vs. 9.59, Goal Compliance 9.75 vs. 9.24.
  • Performance gap between proprietary and open-source models attributed to Multimodal DeepResearcher's higher demands on planning, writing, coding, and refinement capabilities.

Limitations

  • Single baseline (DataNarrative); no comparison to text-only deep research systems on shared metrics.
  • Performance drops substantially with open-source models (55% vs. 82%), indicating strong model-capability dependence.
  • Actor-critic refinement capped at \(N_{\max} = 3\) iterations; quality ceiling set by this hard stop.
  • Relies on a Chrome browser tool for rendering feedback, coupling the pipeline to a specific runtime environment.
  • Ablation results are truncated in the provided text, limiting evaluation of each component's marginal contribution.
  • Exemplar reports are fixed at 6 documents from a narrow set of sources; generalization to other report styles is untested.

Relevance to Harnesses / Meta-Harnesses

Multimodal DeepResearcher is a clear example of a multi-stage, multi-model meta-harness: it orchestrates heterogeneous models (GPT-4o-mini for search/planning, Claude 3.7 Sonnet as both LLM and MLLM) across well-defined pipeline stages, using FDV as a structured inter-stage communication format analogous to a typed message schema between harness components. The actor-critic chart refinement loop is a reusable harness sub-pattern—feedback from a browser executor and MLLM critic drives iterative improvement—directly paralleling the plan-execute-observe loops seen in tool-using agent harnesses. The explicit separation of textualization, planning, and generation stages mirrors harness design principles where each stage produces a typed artifact consumed by the next. For researchers tracking harnesses, this paper is notable for showing how in-context learning from expert exemplars can substitute for fine-tuning when the harness supplies sufficiently rich structured representations between stages.