Skip to content

Multi-Turn Agentic Scientific Literature Search via Workflow Induction

🕒 Published (v1): 2026-07-01 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PAPERPILOT frames scientific literature search as workflow induction: given an anchor paper and user query, the agent constructs an executable DAG of typed paper-search operators (keyword search, citation expansion, filtering, scoring, reranking, evidence extraction) and refines the DAG through multi-turn user feedback. A 9B model trained with supervised workflow imitation followed by IPO-style DPO on corrupted workflow pairs substantially outperforms its base model and rivals much larger systems at orders-of-magnitude lower cost.

Problem

Existing scientific search agents either use fixed retrieval pipelines or represent user intent implicitly in free-form language, making the retrieval strategy opaque and hard to modify. A user's search intent is rarely fully specified in a single query—it is preference-dependent, direction-specific (predecessor/successor/sibling/benchmark/survey), and evolves through interaction—yet current systems treat clarification feedback as additional query text rather than instructions to structurally modify retrieval behavior.

Method

PAPERPILOT defines a PAPERPILOT-TOOLSET: typed operators (keyword search, citation expansion, union/filter/score/rerank, NLI filtering, evidence extraction, workflow construction) that can be composed into type-consistent directed acyclic graphs \(G = (V, E)\), where each node \(v_i = (o_i, \theta_i)\) is an instantiated operator with parameters.

At each turn \(t\), the agent performs a rollout: $\(G_t = \text{Induce}(q, p_0, P_t, H_t), \quad (P_{t+1}, y_t) = \text{Execute}(G_t, P_t)\)$ and then refines the workflow based on user feedback \(f_t\): $\(G_{t+1} = \text{Refine}(G_t, f_t, H_{t+1})\)$

Training proceeds in two phases: 1. Supervised fine-tuning on 5,540 high-quality workflow trajectories generated by a strong teacher model (filtered to turns where the gold paper appears in top-5). 2. Preference optimization using 1,733 hard chosen–rejected pairs, where rejected workflows are created via structural and semantic corruptions (invalid references, missing inputs, incorrect operators, dropped nodes, shifted filters, vague NLI axes); optimized with an IPO-style DPO objective for long-context stability.

Multi-turn evaluation uses a fixed Qwen3.5-397B-A17B simulator with leakage-control (prompt constraints, deterministic string matching, LLM-based leak checker).

Key Contributions

  • Formulation of multi-turn scientific literature search as workflow induction over DAG-structured, executable, user-editable search plans rather than implicit language-level reasoning.
  • PAPERPILOT-TOOLSET: a library of typed paper-search operators enabling compositional, intent-specific retrieval strategies.
  • PAPERPILOT-9B: a 9B model trained via two-phase supervision (workflow imitation + preference optimization on corrupted workflows), achieving 0% workflow execution errors.
  • A reproducible multi-turn benchmark covering five search directions (predecessor, successor, sibling, benchmark, survey) with leakage-controlled user simulation.
  • Workflow-level evaluation metrics (TF-IDF cosine similarity over DAG, edit-consistency scores for add/modify/remove node operations).

Results

  • PAPERPILOT-9B vs. base Qwen3.5-9B PAPERPILOT-TOOLSET (multi-turn): Hit@5 58.0 → 77.0, MRR 47.5 → 59.4, nDCG@10 26.8 → 32.5, workflow execution errors 9.5% → 0%.
  • PAPERPILOT-9B vs. single-turn GPT-5.4 Web Search: surpasses on Hit@5 (77.0 vs. 72.5), Hit@10 (83.5 vs. 79.0), Hit@15 (89.5 vs. 81.0), and MRR (59.4 vs. 60.2 — near parity) at a fraction of the cost.
  • Best overall system: GPT-5.4 + Web Search + PAPERPILOT-TOOLSET (multi-turn): Hit@5 84.0, Hit@10 87.0, Hit@15 89.5, Recall@50 56.8, MRR 71.8, nDCG@10 41.6, at $0.1508/case.
  • Cost comparison: OpenAI DeepResearch costs $6.09/case for 72.0 Hit@5; PAPERPILOT-9B achieves 77.0 Hit@5 at $0.018/case (>300Ă— cheaper).
  • Adaptive vs. fixed workflow (GPT-5.4, single-turn): PAPERPILOT-TOOLSET vs. fixed PAPERPILOT-WORKFLOW: Hit@5 79.5 vs. 52.0, MRR 62.8 vs. 36.4.
  • Workflow generation (ablation): PAPERPILOT-9B whole-DAG TF-IDF cosine 0.362 vs. 0.047 for base Qwen3.5-9B; strongest cost-efficiency among all evaluated systems (Rel. Eff. 3.26 at $0.00108/case).

Limitations

  • Evaluation relies on a single fixed LLM user simulator (Qwen3.5-397B-A17B); real users may exhibit more diverse or adversarial feedback patterns.
  • Gold paper sets are constructed from citation-graph signals and LLM-assisted synthesis rather than fully human-annotated relevance judgments, which may introduce label noise.
  • Recall@50 for PAPERPILOT-9B (40.0) lags behind some larger or fixed-workflow systems that use broader retrieval strategies, suggesting coverage tradeoffs at the tail.
  • The workflow DAG representation requires type-consistent operator composition; errors in type resolution are a documented failure mode for untrained smaller models.
  • Evaluation corpus is limited to Semantic Scholar cached metadata; live retrieval over evolving literature is not assessed.
  • The benchmark is constrained to five predefined search directions; generalization to arbitrary open-ended research tasks is not directly demonstrated.

Relevance to Agentic AI / LLM Agents

PAPERPILOT is a concrete instantiation of the workflow-as-program paradigm for agents: rather than reasoning implicitly in natural language, the agent externalizes its retrieval strategy as an inspectable, editable DAG, enabling genuine feedback-driven control flow modification—a key open challenge in agentic AI. The two-phase training recipe (imitation from a strong teacher + preference optimization over adversarially corrupted execution traces) is a broadly applicable pattern for teaching agents to produce structurally valid plans under multi-turn interaction. The leakage-controlled LLM user simulator methodology directly addresses the evaluation gap in interactive agents, where naive simulation leaks gold information. The cost-efficiency findings reinforce the broader principle that structured tool use with smaller specialized models can match or exceed large general-purpose models for constrained agentic tasks.