Retro-R1: LLM-based Agentic Retrosynthesis¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Retro-R1 is an LLM agent trained end-to-end with PPO to perform multi-step retrosynthetic planning by interacting dynamically with a plug-in single-step retrosynthesis tool. It formulates synthesis planning as sequential decision-making and achieves 55.79% pass@1 success rate on Retro*-190, surpassing the prior state of the art by 8.95%. Strong out-of-domain generalization distinguishes it from classical search-based baselines.

Problem¶

Multi-step retrosynthetic planning requires navigating a vast combinatorial molecular graph, demanding long-horizon strategic reasoning that single-step prediction models and heuristic search algorithms (A*, MCTS) handle poorly, especially on out-of-distribution molecules. Prior LLM applications to retrosynthesis used static QA formats rather than interactive, feedback-driven agents.

Method¶

Retro-R1 wraps a Qwen2.5-7B-Instruct-1M policy in an agentic loop with two action types — CALL (invoke single-step model on a chosen unavailable molecule to get reaction candidates) and SELECT (commit to one candidate, updating the molecular state). The environment enforces format/argument/tool validity checks and returns structured textual feedback. PPO is adapted for this multi-turn setting via: - Environment masking: gradients computed only on agent-generated tokens, not environment responses. - GAE with γ=λ=1: Monte Carlo returns to avoid discount-induced bias from interspersed environment tokens. An iterative planning strategy runs multiple sequential search rounds, maintaining a global synthesis graph (AND-OR graph) with a memory mechanism; revisited states have their candidate orderings shuffled to encourage exploration. Final route extraction finds the shortest path in the accumulated graph.

Key Contributions¶

End-to-end RL training (PPO) of an LLM agent for multi-step retrosynthesis with live tool interaction.
PPO adaptation for agentic multi-turn settings: token-level environment masking and unit-parameter GAE (γ=λ=1).
Iterative planning with memory and context shuffle, building a global synthesis graph across rounds.
State-of-the-art pass@1 (55.79%, +8.95% over prior SOTA) and strongest out-of-domain generalization on ChEMBL-1000.
Data efficiency: trained on only 11,366 routes vs. 299,202 for PDVN, with competitive or superior results.

Results¶

Retro*-190 (in-domain): pass@1 = 55.79% vs. 46.84% (EG-MCTS, second-best); success rate at N=500 = 86.95% vs. 92.11% (PDVN, best at limit); 87 shorter routes vs. 90 (PDVN).
ChEMBL-1000 (out-of-domain): pass@1 = 73.30% vs. 60.00% (PDVN, second-best, +13.3pp); success rate at N=500 = 83.72% vs. 80.60% (PDVN); Retro-R1 is the only method that continues improving success rate for route lengths >20.
Ablation: RL (Retro-R1 V1) outperforms SFT baseline across nearly all metrics; removing context shuffle (Retro-R1-wos) stalls success rate beyond N=100 due to lack of diversity.
Model-agnostic: upgrading the plug-in single-step model from V1→V4 monotonically improves Retro-R1 performance at no retraining cost.

Limitations¶

The agent cannot propose novel reactions; it is fully dependent on the quality and coverage of the plug-in single-step model.
Reaction candidates are selected from high-probability templates only; chemical feasibility in practice is not guaranteed, and reaction conditions are not predicted.
Evaluation uses template-based MLP single-step models—results may not transfer to template-free or graph-based single-step models.
Context shuffle as an exploration mechanism is a heuristic; no principled diversity guarantee.

Relevance to Agentic AI / LLM Agents¶

Retro-R1 is a concrete demonstration of training an LLM agent via RL (PPO) to operate in a structured, tool-augmented environment with real environmental feedback rather than simulated rewards — directly relevant to the active question of how to extend RL-trained reasoning models beyond math/code to scientific domains. The PPO adaptations (environment masking, unit-parameter GAE) are domain-agnostic techniques applicable wherever multi-turn agent trajectories interleave model actions with external tool responses. The iterative planning strategy — accumulating a global graph across multiple search episodes — is a reusable pattern for agentic systems where a single rollout under-explores a combinatorial space. This work also illustrates how plug-in tool modularity enables zero-shot transfer to stronger tools, a key design principle for scalable agentic systems.