Towards Multimodal Data-Driven Scientific Discovery Powered by LLM Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

LLM agents for multimodal data-driven scientific discovery; benchmark and pipeline [ICLR 2026]

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MoSciBench is the first benchmark for multimodal, repository-level, data-driven scientific discovery, comprising 88 end-to-end tasks across 6 domains and 7 modalities, each framed as a cross-modal hypothesis verification workflow. The benchmark is constructed from peer-reviewed publications via a principled four-stage pipeline and reveals that even the strongest agents (o4-mini + ReAct) reach only 48.94% accuracy, with cross-modal alignment accounting for over 60% of failures. Lightweight workflow scaffolding consistently improves accuracy by ~5.7%.

Problem

Existing scientific discovery benchmarks (e.g., ScienceAgentBench, DiscoveryBench) restrict agents to unimodal data and slice-level tasks, failing to capture the reality that discovery requires integrating heterogeneous modalities, reasoning across full repositories, and verifying hypotheses through end-to-end pipelines. No rigorous multimodal testbed existed to expose agent limitations in cross-modal alignment, modeling, and hypothesis-driven reasoning.

Method

Benchmark construction follows a four-stage pipeline applied to peer-reviewed publications: (1) raw data extraction from openly licensed repositories; (2) multimodal processing—feature filtering, multi-source consolidation, unit/timestamp/spatial standardization, and shared-index alignment (e.g., linking satellite imagery to environmental variables via geographic grids); (3) task instruction formulation yielding concise, open-ended hypotheses with explicit answer formats (categorical, numerical, string); (4) human verification via end-to-end executable reference scripts and multi-pass annotation consistency checks (100% agreement enforced).

Evaluation uses exact-match accuracy (Acc), code execution success rate (Exec), and a 1–5 LLM-as-judge Modeling Rationality (MR) score. Six agent frameworks are compared: NoDataGuess (no data, pure LLM), ReAct (reasoning + code execution loop), DataVoyager (modular planner–coder–critic pipeline), Reflexion with oracle feedback (up to 3 retries), SelfDebug (execution-trace-driven self-rewriting), and RAG-ReAct (retrieval-augmented ReAct). All agents run zero-shot at temperature 0.0; per-task compute is capped at 1 hour.

Key Contributions

• MoSciBench: 88 repository-level, cross-modal discovery tasks spanning 6 domains (climate science, biomedical engineering, cheminformatics, health psychology, population genomics, earth science), 7 modalities (multi-sensor time series, tabular, satellite imagery, mass spectra, molecular structures, genotype matrices, HDF matrices), and 5 task categories (descriptive, correlational, causal inference, predictive, pattern discovery).
• Principled construction pipeline: four-stage process from raw peer-reviewed data to verified, executable benchmark instances with gold hypotheses derived from published findings.
• Systematic evaluation of 4 LLM families × 6 agent frameworks, producing structured error taxonomy (alignment, modeling, reasoning) and identifying cross-modal alignment as the dominant failure mode (>60% of errors).
• Scaffolding finding: lightweight workflow scaffolding reduces alignment errors by 5–10% and raises overall accuracy by 5.7% on average.

Results

• Best overall accuracy: 48.94% (o4-mini + ReAct); Reflexion (oracle) achieves 45.8% with same model.
• Smaller models collapse: Qwen3-30B-A3B peaks at 23.3%; gpt-5-mini at 17.4%.
• NoDataGuess baseline (no data access): 0.0% for Qwen3-30B-A3B, 2.6% for DeepSeek-V3.1, 10.5% for o4-mini—confirming data grounding is essential.
• Data-grounded frameworks improve over NoDataGuess by 20–40% across domains.
• Cross-modal alignment failures account for >60% of total errors in the best-performing agent.
• RAG-ReAct with o4-mini reaches 64.3% on health psychology and 58.8% on biomedical engineering (domain-best figures), while earth science and climate science remain hardest (<36% for all agents).
• Workflow scaffolding (e.g., DataVoyager's modular planner–critic structure) reduces alignment errors by 5–10% and lifts accuracy by 5.7% on average over unscaffolded counterparts.
• MR scores cluster between 3.4–4.0 across all settings, indicating agents produce scientifically plausible workflows even when exact-match accuracy is low.

Limitations

• Only 88 tasks total; the scale is deliberately small due to the hours-per-task execution cost, which limits statistical power across the task-type × domain × model × framework grid.
• Exact-match accuracy is strict and may penalize agents that are scientifically correct but differ in precision or answer format from the reference.
• Ground-truth answers are derived from a single peer-reviewed paper per task, introducing potential publication bias; no alternative "correct" solutions are enumerated.
• Agent frameworks evaluated are adaptations of single-domain agents, not purpose-built multimodal discovery harnesses; results may underestimate what a purpose-designed system could achieve.
• Evaluation does not measure efficiency (wall-clock time, token cost) at granular task level; the 1-hour cap is the only budget constraint reported.
• LLM-as-judge MR scores use gpt-4o-mini, introducing a single-model bias in workflow quality assessment.

Relevance to Harnesses / Meta-Harnesses

MoSciBench directly stress-tests agent harness architectures—ReAct, DataVoyager's modular planner–coder–critic pipeline, Reflexion's retry-with-oracle loop, and SelfDebug's execution-trace-driven repair—on realistic, end-to-end multi-step workflows, making it a direct evaluation surface for harness design choices. The finding that lightweight workflow scaffolding reduces alignment errors by 5–10% and raises accuracy by 5.7% is a quantified argument for structured harness stages over flat code-execution loops. The benchmark's four-stage construction pipeline is itself a meta-harness: a reproducible orchestration of data extraction, alignment, annotation, and verification subagents that could serve as a template for automated benchmark creation. For researchers building or comparing scientific discovery harnesses, MoSciBench provides the first multimodal testbed where cross-modal alignment is an explicit, measurable bottleneck, isolating a gap that harness-level interventions (e.g., dedicated alignment stages, retrieval augmentation) must address.