Agentic-Ideation: Sample Efficient Agentic Trajectories Synthesis for Scientific Ideation Agents¶

🕒 Published (v1): 2026-06-30 07:07 UTC · Source: Arxiv · link

Why this paper was selected

Key meta-harness concept: moving beyond fixed workflows via self-evolving trajectories

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Agentic-Ideation is a framework for training an agentic LLM to perform scientific ideation by synthesizing high-quality trajectory data via an oracle-guided multi-agent harness. Instead of rejection sampling over an open-ended answer space, the system injects a "reference idea" as oracle guidance so that a Planner–Controller hierarchy reconstructs logically coherent tool-use trajectories in a single forward pass. The trained 8B model outperforms all workflow-based baselines by 11.91% overall and achieves >10× synthesis efficiency vs. rejection sampling.

Problem¶

Existing scientific ideation systems rely on rigid, hand-engineered workflows (fixed prompt sequences for retrieval → idea generation). Training a flexible agentic LLM instead requires large-scale agentic trajectory data, but standard rejection-sampling data synthesis is incompatible with open-ended ideation: there is no binary "correct answer" ground truth, so unguided rollouts almost never produce a trajectory aligned with any specific high-quality idea, making the acceptance rate prohibitively low.

Method¶

Tool space. Define $\mathcal{T} = \mathcal{T}_{ext} \cup \mathcal{T}_{think}$: three external tools (Search, Get_References, Get_Cited) and three cognitive tools (Analyse_Gap, Ideation, Reflection).

Oracle-guided data synthesis. A hierarchical multi-agent system (Planner + Controller + 6 specialized tool agents) is driven by a "Reference Idea" $I_{ref}$ extracted from the anchor paper. The Controller uses a Dual-Thought Mechanism: at each step $t$ it generates a Private Thought (oracle view, consults $I_{ref}$ to identify the next best action $a_t$) and a Public Thought (researcher view, rationalizes $a_t$ using only the observable history $H_t$, masking $I_{ref}$). Only the Public Thought is retained in the training trajectory, so the synthesized data mimics natural autonomous exploration. A trajectory is kept only when the final generated idea recovers $I_{ref}$ semantically, but because the Private Thought already steers the controller, this succeeds in 1 rollout instead of ~12.

Agentic SFT with result masking. The retained trajectories are linearized with special tokens (<Tool_Name>, <Result>, <Answer>). Loss is computed with a binary mask $M_t$ that zeros out all <Result> tokens: $$\mathcal{L}(\theta) = -\frac{1}{\sum_{t=1}^{|X|} M_t} \sum_{t=1}^{|X|} M_t \cdot \log P_\theta(x_t \mid x_{<t})$$ This forces the model to learn decision-making logic rather than memorizing tool return values.

Key Contributions¶

First application of trained agentic LLMs (as opposed to workflow pipelines) to scientific ideation.
Oracle-Guided Data Synthesis with a Dual-Thought Mechanism that converts open-ended trajectory generation from stochastic search into single-pass directed reconstruction.
10× improvement in synthesis sample efficiency over rejection sampling (1 rollout vs. ~12 per valid trajectory).
SFT result-masking strategy that prevents shortcut learning from external feedback tokens.
State-of-the-art on a 300-paper NeurIPS-2025 benchmark (11.91% overall improvement over best workflow baseline).

Results¶

Overall score: Agentic-Ideation 6.67 vs. best baseline SciPIP 5.97 → +11.91% (p < 0.05).
Novelty: +11.40%; Significance: +11.14%; Feasibility: +8.70% vs. best baselines.
Gains are consistent across NLP (+11.13% overall), CV (+9.67%), and Other (+10.48%) domains.
Synthesis efficiency: Oracle-guided strategy requires 1 rollout/sample vs. rejection sampling's average of 12 rollouts/sample → >10× speedup.
4,646 synthesized trajectories used to fine-tune Qwen3-8B backbone.
Ablation: removing Analyse_Gap causes the largest single Novelty drop; removing Search most impacts Feasibility.

Limitations¶

Oracle guidance requires a high-quality "reference idea" for every training anchor paper — inapplicable when such ground-truth ideas do not exist a priori.
Synthesis uses Qwen3-235B as the oracle agent; inference cost at this scale may not be accessible for all labs.
Evaluation relies on an LLM judge panel (GPT-5.2, Gemini-3-Pro-Preview, Claude-Sonnet-4.5, Qwen3-Max); LLM judges can be miscalibrated for scientific novelty.
Test set is limited to three domains (NLP, CV, Other) from ML conferences; generalization to biology, chemistry, or social sciences is untested.
The masked-result SFT approach assumes tool outputs during inference will be consistent with training distribution; distribution shift in retrieved papers is not analyzed.

Relevance to Harnesses / Meta-Harnesses¶

Agentic-Ideation is a meta-harness for trajectory data synthesis: it wraps a Planner + Controller + 6 specialized tool agents in a coordinated pipeline that produces training data for a downstream model, rather than directly solving a task. The Dual-Thought / oracle-injection pattern is a general harness design technique — injecting privileged global state into an inner agent to steer trajectory quality, then stripping it before training — applicable beyond ideation to any domain where ground-truth outcomes are absent. The masking-during-SFT strategy directly addresses a known failure mode of harness-generated training data (models learning environment noise rather than control logic), making it a reusable design primitive for harness builders.