Skip to content

Autonomous Scientific Discovery via Iterative Meta-Reflection

🕒 Published (v1): 2026-07-01 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DiscoPER is an autonomous scientific discovery framework that runs LLM agents in a PROPOSE–EVALUATE–REFLECT loop over raw multimodal datasets, requiring no pre-specified research questions or seed hypotheses. Its key novelty is a second-order REFLECT module that periodically analyzes the full accumulation of accepted and rejected claims to identify gaps, confounds, and compound hypotheses, redirecting future exploration. On the new iNatDisco ecological benchmark, it recovers 8 of 9 peer-reviewed patterns with a 72.7% held-out validation rate, outperforming all baselines.

Problem

Existing autonomous discovery systems fail in one or more of three ways: (1) classical causal discovery methods (\(H_\text{edge}\)) are limited to pairwise variable edges and cannot express higher-order patterns; (2) code-driven LLM systems (e.g., ExperiGen, HeurekaBench) require externally-supplied research questions or seed hypotheses (\(P \neq \emptyset\)); (3) no prior system performs structured meta-reasoning over its own accumulated claim set to guide subsequent exploration — each hypothesis generation step is effectively independent. Existing benchmarks are also inadequate, evaluating agents on pre-specified questions rather than open-ended pattern discovery.

Method

DiscoPER formalizes discovery as a generalized PROPOSE–EVALUATE–REFLECT loop iterating over a claim set \(C_t\) and guidance \(G_t\):

\[h_t^{(1)}, \ldots, h_t^{(K)} \sim \text{PROPOSE}(X, C_{t-1}, G_{t-1}, P{=}\emptyset)$$ $$\Delta C_t, \Delta\hat{C}_t = \text{EVALUATE}(\{h_t^{(k)}\}_{k=1}^K, X, C_{t-1}, \hat{C}_{t-1})$$ $$G_t = \text{REFLECT}(C_t, \hat{C}_t)\]
  • PROPOSE: An LLM agent receives a tabular schema summary plus optional images, then generates \(K\) hypotheses as natural language + executable Python statistical tests, conditioned on \(C_{t-1}\) and \(G_{t-1}\).
  • EVALUATE: Each hypothesis program runs on a training split; accepted only if effect size \(|\delta| \geq 0.2\), \(p \leq 0.05\), and \(|\delta_\text{val}| \geq 0.6 \cdot |\delta_\text{train}|\) on a held-out validation split. The held-out split is evaluated exactly once (anti-p-hacking). The hypothesis space \(H_\text{code}^\text{open}\) is the set of all Turing-computable statistical tests.
  • REFLECT: A separate LLM agent analyzes the full \(C_t / \hat{C}_t\) pair every 5 iterations, producing structured guidance \(G_t\) identifying under-explored variables, recurring confounds, and compound (higher-order) hypotheses, without access to ground truth.
  • Multimodal: PROPOSE can invoke VLMs (same base LLM) to extract visual attributes from images for hypotheses involving image features.

Key Contributions

  • Unified generalization of existing discovery systems as restricted PROPOSE–EVALUATE–REFLECT instances (Table 1), clarifying the hierarchy \(H_\text{edge} \subset H_\text{code}^\text{guided} \subseteq H_\text{code}^\text{open}\).
  • DiscoPER: first system combining \(H_\text{code}^\text{open}\), \(P = \emptyset\), and explicit structured meta-reflection over accumulated claims.
  • iNatDisco benchmark: iNatDisco-800 (800 observations, 8 species, 9 peer-reviewed ecological patterns) and iNatDisco-50K (50,000 observations, 9,776 species, 12 patterns), both with multimodal (image + tabular) data from iNaturalist.
  • iNatDisco-800-CF: counterfactual variant with five patterns deliberately reversed in tabular data (images unchanged), used to verify that the system discovers from data rather than LLM memorization.

Results

  • iNatDisco-800: DiscoPER recovers 8/9 patterns (recall), support rate 72.7% ± 3%. Best baselines: ExperiGen and HeurekaBench-like both achieve 3/9; all classical causal methods recover ≤1/9.
  • iNatDisco-50K: DiscoPER recovers 8/12 patterns, support rate 74.2% ± 3%. ExperiGen-like: 3/12.
  • Ablation (no REFLECT): iNatDisco-800: 7/9; iNatDisco-50K: 6/12 — REFLECT provides a consistent +1–2 pattern lift and higher support rate on both.
  • Data scaling: recall improves from ~17% at \(n=800\) to ~75% at \(n=50\)K; support rate also rises, indicating larger datasets allow more reliable grounding.
  • Iteration scaling: recall increases monotonically with more iterations; support rate decreases (from ~98% at 10 iterations to ~67% at 100) as the model exhausts easy hypotheses and shifts to speculative ones.

Limitations

  • As iterations increase, a growing fraction of PROPOSE outputs are grounded in LLM internal knowledge rather than the data (visible in counterfactual analysis), suggesting the exploration degrades toward memorization over long runs.
  • Support rate declines with more iterations, indicating an inherent trade-off between discovery breadth and hypothesis precision.
  • 4 ground-truth patterns in iNatDisco-50K remain unrecovered even at 100 iterations with full data; the text (truncated) does not fully characterize these failure modes.
  • Held-out anti-p-hacking constraint (single evaluation pass) limits iterative code refinement, potentially discarding valid hypotheses with imperfect initial implementations.
  • Computational cost of 100-iteration runs with multimodal VLM calls is not characterized.

Relevance to Agentic AI / LLM Agents

DiscoPER is a concrete instantiation of a multi-agent loop where one agent generates executable actions (hypotheses as code), a code-execution evaluator grounds those actions in real data, and a second-order reasoning agent synthesizes the trajectory of prior actions to steer future ones — a direct analog of reflection and self-correction mechanisms studied in LLM agent research. The REFLECT module operationalizes "memory over agent trajectories" beyond simple context concatenation, treating prior outputs as empirical data to reason about. The anti-hallucination design (every claim must pass statistical tests on held-out data) addresses a core reliability challenge in agentic systems that produce factual claims. The work also demonstrates tool use (VLM calls from within the agent loop) and scales the agentic loop empirically, providing insight into how recall vs. precision trade-offs evolve with iteration count.