Skip to content

Towards Verifiable Multimodal Deep Research: A Multi-Agent Harness for Interleaved Report Generation

🕒 Published (v1): 2026-05-28 12:40 UTC · Source: Arxiv · link

Why this paper was selected

Multi-agent harness for verifiable interleaved multimodal deep research; directly named topic

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PTAH is a multi-agent harness that orchestrates verifiable multimodal deep research—from a plain-text query to a rendered HTML report—through Planning, Research, and Writing stages with a Verifier Agent serving as a stage-gating acceptance function. It introduces Visual Working Memory to maintain source-aligned image candidates as inspectable intermediate state rather than treating images as post-hoc decoration. Accompanying it is PTAHEval, an evaluation protocol that adds image-level (ICQ) and presentation-level (MPQ) assessment on top of existing deep research benchmarks.

Problem

Existing deep research pipelines have two critical gaps: (1) they lack stage-wise verification, allowing noise introduced early to propagate and produce factually unreliable outputs; (2) they treat image integration as a decorative post-processing step, producing visual evidence that is loosely coupled from textual arguments. No open-source framework addresses both jointly, and existing evaluation metrics are text-only and insufficient for interleaved image–text reports.

Method

PTAH formalizes the research process as a harnessed agentic trajectory. At each step \(t\), the harness maintains research state \(s_t = (q, M_t, \tau_{<t})\), where \(M_t\) is a structured working state holding intermediate plans, evidence, citations, numerical data, and visual candidates; \(\tau_{<t}\) is the interaction history. Tool invocations \(a_t = (u_t, x_t)\) yield observations \(o_t = u_t(x_t)\) that update \(M_{t+1}\). The final report \(r \sim p_\theta(\cdot \mid q, M_T, \tau)\) is serialized to HTML and rendered.

Three stages: - Planning: A Planner Agent iteratively searches and produces a structured plan specifying textual sections, section-level research goals, and visual specifications (placement, communicative role, visual form—chart/screenshot/diagram). The Verifier Agent checks plan validity via rule-based protocol validation and LLM-based rubric scoring before advancing. - Research: Parallel Researcher Agents each handle one section independently, producing claim-grounded evidence packages (findings, numerical data, tables, citations, writing instructions). Each also extracts images from visited pages into a per-section Visual Working Memory (VWM)—filtered by resolution and relevance with a VLM selector, stored with source URL, page context, assigned section, and intended visual role. The Verifier Agent checks each package for citation accuracy, goal coverage, and visual relevance before it advances. - Writing: A Writer Agent performs declarative multimodal composition, embedding image tool tags directly in the text at intended positions. The harness arbitrates three image operations: Image Reference (reuse from VWM), Image Search (retrieve additional web images), Image Generation (synthesize charts/diagrams via code execution or generative models). Test-time scaling then applies six sequential lifecycle refinement hooks: Section Refine → Image Refine (Keep/Delete/Edit per image) → Overall Refine → HTML Generate → HTML Refine → Render.

Implementation uses Qwen3-32B as Planner/Researcher/Verifier and Qwen3-VL-32B-Instruct as Writer; Qwen3-VL-235B-A22B-Instruct serves as VLM judge in PTAHEval.

PTAHEval augments existing benchmarks with two evaluation dimensions, each scored 1–5 Likert: ICQ (Visual Clarity, Cross-Modal Alignment, Information Complementarity, Evidentiary Support) and MPQ assessed on a 1000×2000 pixel viewport screenshot (Density-Legibility Balance, Informational Saliency, Visual Encoding Diversity, Visual Ergonomics).

Key Contributions

  • PTAH harness: explicit three-stage lifecycle (Plan/Research/Write) with a Verifier Agent as stage-gating acceptance function combining rule-based and LLM-based rubric checks
  • Visual Working Memory: structured cross-modal intermediate state that preserves source-aligned image candidates with provenance metadata for downstream composition
  • Declarative multimodal composition: writer embeds image directives inline rather than selecting images post-hoc; harness arbitrates Reference/Search/Generate operations
  • Verifier hooks for cross-modal consistency: enforce factual grounding, citation fidelity, visual relevance, and image–text alignment at each workflow boundary
  • PTAHEval: evaluation protocol adding ICQ + MPQ dimensions to existing deep research benchmarks, enabling assessment of interleaved report quality

Results

  • DeepResearch Bench (overall text score): PTAH 45.16 vs. WebThinker 45.00 (best text-only baseline), leading on Insight/Depth (44.32 vs. 43.26) and Readability (47.95 vs. 46.86)
  • DeepConsult (average): PTAH 16.18 vs. WebThinker 7.35 (best baseline)—more than 2Ă— improvement; largest gains on Instruction-Following (13.73 vs. 2.94), Completeness (17.64 vs. 17.64 tie with WebThinker), Writing Quality (14.71 vs. 5.88)
  • ICQ average (DeepResearch Bench): PTAH 4.39 vs. LLM-I 1.97 (only multimodal baseline); CMA 4.79, VC 4.42, IC 4.35, ES 4.01
  • MPQ average: PTAH 3.71 vs. best non-multimodal baseline QwQ-32B 3.45; text-only baselines (ReAct/Search-o1/WebThinker) score ~3.08–3.12 due to absence of images
  • Citation Accuracy: 87.53 with 9.64 effective citations per task, substantially outperforming all baselines; residual errors stem from minor content-source mismatches

Limitations

  • No readily reproducible open-source general multimodal research report generation framework exists for comparison; LLM-I is the only multimodal baseline and is narrower in scope
  • VLM-as-judge evaluation (Qwen3-VL-235B) may introduce model-specific biases in ICQ/MPQ scoring
  • Residual citation errors remain from minor mismatches between cited content and source pages
  • Text-only benchmark scores (DeepResearch Bench overall) show only marginal improvement over WebThinker (45.16 vs. 45.00), suggesting the textual quality gains are modest relative to the system's added complexity
  • Paper is truncated; explicit ablation studies and failure-mode analyses are not fully visible

Relevance to Harnesses / Meta-Harnesses

PTAH is a first-class example of a multi-agent harness as an architectural pattern: it explicitly names itself a harness, defines a stage-gating acceptance function (Verifier Agent), maintains inspectable intermediate state (Visual Working Memory, structured research packages), and uses lifecycle hooks (six test-time refinement steps) rather than monolithic generation. The Verifier-as-acceptance-function design—combining rule-based protocol checks with LLM rubric evaluation at each stage boundary—is a concrete instantiation of how harnesses enforce contracts across agent handoffs. For researchers tracking this space, PTAH demonstrates how to extend the harness pattern into multimodal workflows while preserving the core harness properties: deterministic stage advancement conditions, observable intermediate artifacts, and composable specialized sub-agents.