Skip to content

IRIS: Interactive Research Ideation System for Accelerating Scientific Discovery

🕒 Published (v1): 2025-04-23 14:01 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IRIS is an open-source, human-in-the-loop scientific ideation platform that orchestrates three specialized agents (ideation, review, retrieval) via Monte Carlo Tree Search to iteratively explore and refine research hypotheses. It addresses the critical gap where fully automated systems produce "objectively novel" yet goal-misaligned ideas by embedding researcher intervention at every stage. A user study with 8 researchers across disciplines validates consistent hypothesis quality improvement.

Problem

Existing LLM-based hypothesis generation systems generate ideas in a single pass, use coarse-grained criteria (novelty, rigor) as feedback without targeting specific components, employ simplistic keyword-based retrieval, and explore the idea space either through pure exploitation (refinement of one base idea) or pure exploration (search without refinement). These designs exclude meaningful human steering, enabling LLM "reward hacking" — e.g., adopting fancy terminology or gratuitous graph proposals to game LLM-as-a-judge metrics without genuine scientific value.

Method

IRIS uses a three-agent architecture driven by MCTS:

  • Ideation Agent: Operates in semi-automatic mode (researcher-guided) or fully autonomous mode, generating and refining a research brief \(B\) = {Title, Methodology, Experiment Plan} from a research goal \(G\).
  • Review Agent: Auto-triggered after each generation, scoring the brief against a hierarchical taxonomy grounded in peer-review practice. When explicitly triggered, it switches to fine-grained segment-level evaluation, assigning targeted feedback to distinct brief components. The researcher verifies and filters feedback before it contributes to the reward signal, directly preventing reward hacking.
  • Retrieval Agent: Generates targeted queries for the Ai2 Scholar QA API (Semantic Scholar corpus, >200M papers), applies two-stage retrieval (snippet search → re-ranking), then three-stage generation (quote extraction → cluster-based section planning → cited report generation). Supports researcher-uploaded PDFs parsed via Grobid.

MCTS Adaptation: The search tree \(T\) is rooted at \(G\); each node stores state \(s = \{b, r, f, k\}\). The action space \(A = \{a_1\text{: generate}, a_2\text{: refine w/ retrieval}, a_3\text{: refine w/ review}, a_4\text{: refine w/ user feedback}\}\). Node selection uses UCT: $\(\text{UCT}(n) = \frac{Q(n)}{N(n)} + c\sqrt{\frac{\ln N(n_p)}{N(n)}}\)$ where \(Q(n)\) is accumulated reward, \(N(n)\) visit count, and \(c\) controls exploration. Budget constraints lower \(c\) to prioritize exploitation under cost limits. Agents maintain trajectory-level memory to steer toward non-redundant refinements.

Key Contributions

  • Human-in-the-loop framework embedding researcher control at generation, feedback verification, and retrospection stages.
  • MCTS-based adaptive test-time compute expansion for systematic exploration/exploitation of the hypothesis space.
  • Fine-grained hierarchical review taxonomy enabling segment-level, actionable feedback that mitigates reward hacking.
  • Query-based multi-document retrieval with clustering, re-ranking, and cited section-wise reports.
  • Open-source platform (HTML/CSS/JS frontend, Gemini-2.0-Flash via LiteLLM, swappable LLM backend).

Results

  • LLM-as-a-judge absolute scores improved by +0.5 points at MCTS tree depth 3 vs. initial generation.
  • ELO ratings improved by +12 points at tree depth 3.
  • Human-LLM ELO correlation: Pearson's \(r = 0.60\); absolute score correlation: \(r = 0.45\) (motivating shift to ELO as primary metric).
  • User ratings (N=10, 1–5 Likert): Fine-grained feedback 4.3 ± 0.7; MCTS steerability 4.2 ± 0.6; Usability/control 4.5 ± 0.7; Overall satisfaction 3.9 ± 0.7; Literature summaries 3.7 ± 0.8.
  • Post-interaction: 25% found hypotheses substantially better, 50% marginally better, 25% similar; 100% reported enhanced methodological understanding.
  • 87.5% of users found critiques reflected their own concerns; 62.5% found outputs relevant to ongoing work.

Limitations

  • Assumes sufficient researcher domain expertise to act as judge; breaks down for non-expert users.
  • Not evaluated with frontier reasoning models (Claude 3.7 Sonnet, Grok-3, Gemini-2.5-Pro, o1) due to budget constraints.
  • Retrieval quality degrades for domains with sparse Semantic Scholar coverage (chemistry/physics rated 3.7/5 vs higher for AI/NLP).
  • MCTS is computationally intensive; budget controls are a workaround, not a structural solution.
  • Small user study (\(N=8\), 10 sessions); limited statistical power for quantitative claims.

Relevance to Harnesses / Meta-Harnesses

IRIS is a concrete meta-harness instantiation: it coordinates three specialized agents through a structured MCTS orchestration loop, where the harness (not any single agent) decides which action to take next, allocates compute across exploration/exploitation phases, and manages memory and budget at the system level. The MCTS loop is directly analogous to a meta-harness control loop — it selects, evaluates, expands, and backpropagates, governing lower-level agent invocations. The human-in-the-loop integration point — where the researcher verifies feedback before it becomes a reward signal — represents a form of meta-level governance that prevents agent-level reward hacking, a design pattern relevant to any harness that uses LLM-as-a-judge as an inner reward. The open-source implementation and explicit budget/exploration controls make it a reusable harness template for adaptive, multi-agent agentic pipelines.