Skip to content

RetroAgent: Harnessing LLMs to Search Over Structured Memory for Agentic Retrosynthesis Planning

🕒 Published (v1): 2026-07-16 03:05 UTC · Source: Arxiv · Venue: COLM 2026 · link

Why this paper was selected

COLM 2026; structured memory for LLM agentic planning in complex combinatorial search spaces

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RetroAgent is an LLM agent that performs multi-step retrosynthesis planning by coupling a language model with a persistent AND-OR graph search structure via tool calls, trained end-to-end with outcome-based RL. It achieves the highest pass@1 on the standard USPTO-190 benchmark using a 4B-parameter model (smaller than the 7B prior state-of-the-art), and surpasses all methods on the out-of-distribution ChEMBL-1000 benchmark by 5.3% in pass@1.

Problem

Multi-step retrosynthesis — decomposing a target molecule into commercially available building blocks through a sequence of valid reactions — has a combinatorial search space that is intractable even for expert chemists. Classical planners rely on value functions trained independently as proxy objectives (Retro*, PDVN), scoring molecules in isolation without conditioning on full route context. Recent LLM-based approaches (Retro-R1) provide only a two-action interface that forces depth-first commitment to one reaction per molecule, limiting exploration of alternative decompositions. No prior method gives an LLM agent direct, flexible control over a coherent global search state.

Method

RetroAgent maintains an AND-OR graph \(G = (V^M \cup V^R, E)\) as a structured memory that persists across interaction turns: molecule nodes \(V^M\) have OR semantics (solved if any child reaction is solved) and reaction nodes \(V^R\) have AND semantics (solved only if all child molecules are solved). The environment handles status propagation (\(O(d)\) upward traversal where \(d\) is route depth), cycle detection, and frontier computation automatically after each tool call.

The agent interacts with this graph through two tool categories: - Memory tools: retrieve candidate reactions for a frontier molecule, apply one or multiple reactions in a single call (multi-expand), and inspect the updated frontier. - Chemistry tools: query synthetic accessibility scores and building-block availability for heuristic candidate prioritization.

The key design decisions are: (1) multi-expand — the agent chooses how many of the top-\(k\) candidate reactions to apply per call, interpolating between single-expand (Retro-R1) and exhaustive-expand (Retro*); (2) environment-side validity enforcement — cycle detection and duplicate filtering are offloaded from the LLM to the environment; (3) building-block annotations in the single-step model's output, giving the agent immediate proximity-to-completion signal without extra tool calls.

The agent policy \(\pi_\theta\) (Qwen3-4B-Instruct-2507) is trained end-to-end via GSPO (sequence-level GRPO with clipped surrogate loss) using an outcome-based reward:

\[r = \begin{cases} 1.0 - \delta_N - \delta_D + p, & \text{if route completed} \\ -1.5, & \text{if no valid tool call ever made} \\ -1.0, & \text{otherwise} \end{cases}\]

where \(\delta_N = \min(0.3,\, 0.02 \cdot \max(0, N - 20))\) penalizes excessive single-step model calls and \(\delta_D\) penalizes route depth beyond 20, each capped at 0.3. Advantages are normalized within groups of \(G=8\) rollouts per target molecule, and Truncated Importance Sampling reuses each batch for 2 gradient steps.

Key Contributions

  • Structured-memory harness that couples an LLM with a persistent AND-OR search graph, cleanly separating deterministic state management from strategic reasoning.
  • Multi-expand tool interface allowing the agent to apply one or more reactions per call, learning its own branching factor rather than following a fixed policy.
  • Environment-enforced validity: cycle detection, duplicate filtering, and status propagation handled outside the LLM, ensuring the agent never sees chemically invalid candidates.
  • End-to-end RL with GSPO on verifiable outcome-based rewards, with shaped penalties for budget overuse and route depth.
  • Strong out-of-distribution generalization: the harness induces transferable planning strategies rather than dataset-specific memorization.

Results

  • USPTO-190 (in-distribution):
  • RetroAgent: 53.26% pass@1, 85.05% at \(N=500\) — best pass@1 across all methods.
  • Retro-R1 (7B): 50.00% pass@1, 84.32% at \(N=500\) — RetroAgent uses a 1.75× smaller model.
  • PDVN: 42.63% pass@1, 92.11% at \(N=500\) — surpasses LLM methods at high budgets by exhaustively applying all predictions (up to 50) per expansion vs. RetroAgent's mean of 1.71 reactions per expand call.
  • ChEMBL-1000 (out-of-distribution):
  • RetroAgent: 73.82% pass@1, 84.40% at \(N=500\) — best on all metrics.
  • Retro-R1: 68.52% pass@1 (+5.30% gap), 82.42% at \(N=500\).
  • PDVN: 60.00% pass@1, 80.60% at \(N=500\) — surpassed by both LLM methods.
  • Ablations (USPTO-190):
  • Zero-shot (no RL): 4.42% pass@1 — RL training is essential; the structured interface alone is insufficient.
  • Without multi-expand: 40.63% pass@1 (−12.6%).
  • Without depth penalty: 51.16% pass@1 (−2.1%), also produces deeper routes (mean depth 31.4 vs. 38.3 steps without penalty).
  • Without both: 39.58% pass@1.

Limitations

  • Template coverage ceiling: the single-step model predicts over 381,302 USPTO reaction templates; novel reactions outside this library are unreachable regardless of search quality.
  • High-budget disadvantage: at \(N \geq 400\), exhaustive search methods (PDVN 92.11% at \(N=500\)) outperform because they explore a larger fraction of the search space per budget unit — the comparison is structurally unequal.
  • Context length scaling: LLM-based methods cannot exhaust the budget in one trajectory due to long-context limitations, requiring the iterative multi-trajectory protocol, which complicates direct comparison.
  • Proprietary Retro-R1 weights: variants V2–V4 (likely stronger) were excluded due to chemical validity concerns or unavailability, making the LLM baseline comparison potentially conservative.
  • Deliberate no-thinking model: Qwen3-4B without chain-of-thought reasoning is chosen to isolate harness contribution, leaving open how much additional gain thinking capabilities would add.

Relevance to Agentic AI / LLM Agents

RetroAgent is a concrete instantiation of the principle that LLM agents perform better when the environment enforces state consistency and the agent focuses purely on strategy — a pattern with direct implications beyond chemistry for any long-horizon planning task. The architecture cleanly separates the environment (graph validity, propagation, frontier computation) from the agent (which molecule to expand, how many reactions to apply), which is a transferable design template for agentic systems operating over structured state. The finding that a 4B model with a well-designed harness outperforms a 7B model with a weaker interface directly challenges the assumption that raw model scale drives agentic performance. The end-to-end RL training with verifiable outcome-based rewards (no learned value function, no intermediate supervision) also connects to broader trends in training agents with GRPO/GSPO on tool-use trajectories.