MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses¶

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MOOSE-Chem is a multi-agent LLM framework that decomposes chemistry hypothesis discovery into three executable subtasks—inspiration retrieval, hypothesis composition, and hypothesis ranking—derived from a formal probabilistic decomposition of \(P(h \mid b)\). Applied to TOMATO-Chem, a 51-paper benchmark of Nature/Science-level chemistry papers published after January 2024, it rediscovers many ground-truth hypotheses with high similarity, free of data contamination. The work also proposes that frontier LLMs may already encode latent scientific knowledge associations unknown to human researchers.

Problem¶

Prior LLM-based scientific discovery systems either operate only in narrow domains (catalyst search, social science, NLP), rely on evaluations prone to data contamination, or treat \(P(h \mid b)\) as a monolithic generation problem—making it practically intractable for open-domain chemistry where hypotheses require multi-hop, cross-domain inspiration synthesis.

Method¶

The core insight is a probabilistic factorization of hypothesis discovery:

\[P(h \mid b) \approx \prod_{j=1}^{k} P(i_j \mid b, h_{j-1}, \mathcal{I}) \cdot P(h_j \mid b, i_j, h_{j-1}), \quad h_0 = \emptyset\]

with a separate ranking function \(R(h) \to \mathbb{R}\) for post-hoc ordering. MOOSE-Chem implements this as a three-stage multi-agent pipeline:

Inspiration retrieval \(P(i_j \mid b, h_{j-1}, \mathcal{I})\): LLM screens the literature corpus \(\mathcal{I}\) in sliding windows of 15 papers, selecting 3 candidate inspirations per window; multiple screening rounds prune the corpus exponentially (top 20% → 4% → 0.8%).
Hypothesis composition \(P(h_j \mid b, i_j, h_{j-1})\): An "evolutionary unit" (EU) generates multiple association "mutations," refines each with LLM feedback on validity/novelty/clarity/significance, then recombines survivors—a mini evolutionary algorithm over hypothesis space.
Ranking \(R(h)\): An LLM scores each hypothesis on validity, novelty, significance, and potential; beam search (beam size 15, default 3 rounds) selects top-\(k\) hypotheses to carry into the next round.

The benchmark TOMATO-Chem consists of 51 chemistry/materials science papers from Nature, Science, and equivalent journals, each annotated by chemistry PhD students with background, ground-truth inspirations, and hypothesis. All papers went online after January 2024; GPT-4o (cutoff October 2023) is used, ensuring no data contamination.

Key Contributions¶

First formal probabilistic decomposition of open-domain chemistry hypothesis discovery into practical, executable subtasks.
MOOSE-Chem framework directly implementing the decomposition with evolutionary hypothesis composition and multi-round beam-search inspiration retrieval.
TOMATO-Chem benchmark: 51 Nature/Science-level chemistry papers annotated with backgrounds, inspirations, and hypotheses; includes a ranking evaluation task absent from prior work.
Empirical evidence that GPT-4o achieves >83% hit ratio on ground-truth inspirations within the top 4% of a 3,000-paper corpus—suggesting LLMs encode latent knowledge associations not yet recognized by scientists.
Demonstration that an LLM-based framework can rediscover the core innovations of Nature/Science chemistry hypotheses without data contamination.

Results¶

Inspiration retrieval (Q1): With a 300-paper corpus, GPT-4o achieves 96.7% hit ratio in the top 20% and 83.7% in the top 4%; even at top 0.8%, hit ratio is 60.8%.
Corpus scaling: Hit ratios remain stable across corpus sizes 150–3,000; top-4% hit ratio ranges from 76.8% (150 papers) to 88.9% (1,000 papers).
Window size ablation: Smaller screening windows outperform larger ones; window=10 yields 98.0% hit ratio in round 1 vs. 71.6% for window=60 at matched corpus size.
Model comparison: GPT-4o (83.7% top-4%) and Llama-3.1-405B (78.7%) substantially outperform Llama-3.1-8B (43.5%) on inspiration retrieval.
End-to-end: MOOSE-Chem rediscovers many ground-truth hypotheses with high similarity scores (exact numbers for final hypothesis similarity truncated in provided text).

Limitations¶

Benchmark covers only 51 papers; the difficulty distribution and coverage of chemistry subfields may not be representative.
Inspiration retrieval assumes LLM screening windows are sufficient—very large corpora (\(>\)3,000 papers) increase multi-round complexity exponentially.
Hypothesis composition relies on LLM self-evaluation for ranking; the paper notes LLMs may not be reliable enough as direct quality judges in chemistry.
The evolutionary unit's recombination step and beam size are fixed heuristics without theoretical optimality guarantees.
Only chemistry and materials science are evaluated; generalizability to other natural science domains is unverified.
Verification of generated hypotheses still requires human experimental validation; the framework produces ranked candidates, not verified results.

Relevance to Harnesses / Meta-Harnesses¶

MOOSE-Chem is a direct instance of a domain-specific agentic harness: it orchestrates multiple LLM inference calls through a structured, multi-stage pipeline (screen → mutate/refine/recombine → rank) with explicit control flow (beam search, fixed rounds, sliding windows) that the harness—not the model—governs. The evolutionary unit pattern—generate mutations, apply LLM feedback, recombine—is a reusable harness primitive for iterative self-refinement that appears broadly in meta-harness designs. The probabilistic decomposition framework offers a principled template for breaking any intractable generation task into harness-schedulable subtasks, relevant to harness architects designing pipelines for open-ended scientific or creative generation. The benchmark construction methodology (contamination-controlled, expert-annotated, with structured component decomposition) also models best practices for evaluating harness pipelines end-to-end.