Towards Multimodal Data-Driven Scientific Discovery Powered by LLM Agents¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Multi-modal data-driven scientific discovery benchmark for LLM agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MoSciBench is the first benchmark for multimodal, repository-level, data-driven scientific discovery, built from 88 tasks derived from peer-reviewed studies across 6 domains and 7 data modalities. It reveals that even the strongest LLM agent (o4-mini + ReAct) achieves only 48.94% accuracy, with cross-modal alignment accounting for over 60% of failures. Lightweight workflow scaffolding consistently reduces alignment errors by 5–10% and raises accuracy by 5.7% on average.

Problem¶

Existing scientific discovery benchmarks (ScienceAgentBench, DiscoveryBench) restrict agents to unimodal datasets and slice-level tasks, missing the cross-modal integration, full-repository access, and hypothesis-driven reasoning that characterize real scientific workflows. There is no rigorous testbed for evaluating agents on end-to-end multimodal discovery.

Method¶

MoSciBench is constructed via a four-stage principled pipeline applied to peer-reviewed publications: 1. Raw data extraction — multimodal datasets (imagery, time series, tabular, molecular structures, mass spectra, genotype matrices, HDF matrices) extracted from papers with permissive licenses. 2. Multimodal processing and alignment — feature filtering, multi-source consolidation, unit/timestamp/spatial standardization, and shared-index alignment (e.g., subject IDs linking physiological signals to tabular attributes; geographic grids linking satellite imagery to environmental variables). 3. Task instruction formulation — each task encodes a background context, a verifiable hypothesis, and a strict answer format (categorical, numerical, string, or structured) without prescribing solution steps. 4. Human verification and quality control — annotators run end-to-end executable scripts against gold-standard hypotheses; tasks with any inconsistency are filtered, yielding 100% human–gold agreement.

Tasks are framed as cross-modal hypothesis verification workflows: agents must autonomously load, preprocess, align, model, and reason over heterogeneous repositories before emitting a verifiable answer. Evaluation uses exact-match accuracy (Acc), code execution success rate (Exec), and a 1–5 LLM-as-judge modeling rationality score (MR, judged by gpt-4o-mini). Six agent frameworks are evaluated: NoDataGuess, ReAct, DataVoyager, Reflexion (Oracle), SelfDebug, and RAG-ReAct, each combined with Qwen3-30B-A3B, DeepSeek-V3.1, gpt-5-mini, and o4-mini at temperature 0.0 zero-shot.

Key Contributions¶

MoSciBench benchmark: 88 end-to-end tasks spanning 6 scientific domains (climate science, biomedical engineering, cheminformatics, health psychology, population genomics, earth science), 7 modalities, and 5 discovery question types (descriptive, correlational, causal inference, predictive modeling, pattern discovery), derived from peer-reviewed studies.
Cross-modal hypothesis verification formalism: tasks require full repository access and multimodal alignment as a prerequisite to modeling and reasoning, contrasting with prior slice-level or unimodal benchmarks.
Systematic multi-agent, multi-LLM evaluation: 4+ agent frameworks × 4 LLM families revealing that cross-modal alignment is the dominant failure mode (>60% of errors) and that workflow scaffolding is consistently beneficial.

Results¶

Best overall accuracy: o4-mini + ReAct = 48.94%; o4-mini + SelfDebug = 48.94%; o4-mini + Reflexion = 45.8%.
Non-data-driven baseline collapse: NoDataGuess achieves 0.0% (Qwen3-30B-A3B), 2.6% (DeepSeek-V3.1), 10.5% (o4-mini); data-grounded frameworks improve by 20–40% across domains.
Smaller models substantially weaker: Qwen3-30B-A3B best overall 34.8% (SelfDebug); gpt-5-mini best 17.4%.
Error distribution: >60% of failures attributed to cross-modal alignment errors; modeling and reasoning errors are secondary.
Scaffolding effect: lightweight workflow scaffolding (e.g., SelfDebug, RAG-ReAct) reduces alignment errors by 5–10% and raises average accuracy by 5.7%.
Domain variance: Biomedical Eng. easiest for strong models (o4-mini + SelfDebug 64.7%); Earth Sci. and Cheminformatics hardest (o4-mini + ReAct 7.1% and 33.3% respectively for some configs).

Limitations¶

Benchmark contains only 88 tasks; coverage per domain is 13–17 tasks, limiting statistical power for fine-grained analysis.
Exact-match accuracy is strict and may undercount partially correct numerical answers despite per-task tolerances.
Evaluation is zero-shot only; few-shot or fine-tuned agents are not studied.
No purpose-built multimodal discovery agent exists; all baselines are adaptations of single-domain frameworks.
Maximum 1-hour per-task compute budget may favor fast but shallow approaches over thorough iterative ones.
Human verification by annotators who wrote the tasks may introduce confirmation bias despite quality controls.

Relevance to Agentic AI / LLM Agents¶

MoSciBench directly stress-tests the core agentic loop—plan, tool-use, data integration, iterative debugging—under realistic scientific complexity, revealing that cross-modal alignment (not reasoning or code generation) is the binding constraint at the frontier of agent capability. The consistent benefit of workflow scaffolding (SelfDebug, RAG-ReAct) supports the broader finding that structured agent orchestration and self-repair loops are more effective than naive chain-of-thought. For researchers building or evaluating LLM agents, this benchmark establishes that unimodal proxies substantially overestimate readiness for real-world scientific tasks, and that repository-level, multi-file, multi-modality settings expose qualitatively different failure modes. It also motivates investment in modality-aware preprocessing and alignment modules as first-class components of agentic scientific pipelines.