Agentic generation of verifiable rules for deterministic, self-expanding reaction classification¶

🕒 Published (v1): 2026-07-01 15:24 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents a fully automated multi-agent LLM pipeline that classifies 665,901 chemical reactions from the USPTO corpus and auto-generates verifiable SMIRKS reaction rules, expanding a seed taxonomy of 68 classes to 14,073 without human curation. A five-agent architecture with a Chain-of-Verification design loop iteratively drafts, tests, and refines SMIRKS patterns until false-positive rates converge to zero. The resulting hybrid classifier achieves 97.7% strict-match accuracy, matching proprietary tools while enabling on-demand extension to novel chemistry.

Problem¶

Existing reaction classification tools (NameRXN, Rxn-INSIGHT) rely on static, manually curated ontologies and hand-authored SMIRKS templates that cannot accommodate novel chemistry, mis-assign rare transformations, or scale to the long-tail distribution of chemical reactions. No prior automated method had produced a validated, large-scale library of generalized, deterministic reaction SMIRKS.

Method¶

Multi-agent taxonomy expansion: Five specialized LLM agents operate in sequence on template-level cohorts (reactions sharing a retrosynthetic template extracted by RDChiral): (1) a high-level classifier maps cohorts to one of 68 seed classes; (2) a detailed classifier refines to the finest applicable level; (3) a verifier agent checks assignments against reaction SMILES—accepting or rejecting; (4) a generator agent proposes new taxonomy entries for rejected reactions; (5) an aggregator/taxonomy-updater deduplicates proposals and atomically extends the living hierarchy. This follows the Chain-of-Verification paradigm to reduce hallucination.

Iterative SMIRKS generation: For each taxonomy class, an LLM drafts broad SMIRKS (high recall, low specificity) in an 80/20 train/test split. Candidate SMIRKS are screened against the full corpus to identify false positives. False-positive examples and accuracy metrics are fed back to the LLM as qualitative feedback in a five-stage refinement loop; the LLM applies chemically grounded mutations (tighten H-count, add context atoms, merge halides, enforce ring membership) until false positives are eliminated without sacrificing recall.

Hybrid deterministic classifier: At inference time, a lightweight MLP classifier (trained on reaction fingerprints) predicts a high-level class, then a topologically ordered template database (ordered via Tarjan's SCC algorithm on a false-positive interference graph) is checked for deterministic SMIRKS match. An LLM fallback classifies and proposes new taxonomy entries for reactions that escape existing templates.

Label noise is estimated via the confident learning framework using out-of-fold MLP probabilities, and cross-taxonomy agreement with NameRXN is measured via greedy modal mapping and adjusted mutual information (AMI).

Key Contributions¶

A self-expanding, living reaction taxonomy grown from 68 → 14,073 classes over 665,901 USPTO reactions with no human curation
First demonstrated automated generation of syntactically valid, chemically precise SMIRKS by LLMs, validated at scale
A five-agent Chain-of-Verification harness that decomposes classification into generation, verification, and dynamic ontology update
An iterative SMIRKS refinement loop with corpus-level false-positive feedback converging to zero FP
Template ordering via false-positive interference graph (Tarjan's SCC + topological sort) for deterministic assignment priority
Estimated true mislabelling rate of ~0.33% (vs. 0.59% for NameRXN under the same procedure)

Results¶

97.7% strict-match accuracy on unseen reactions at hierarchical level 3 using the hybrid classifier
Label noise: LLM pipeline 2.19% raw estimated noise; after LLM-as-judge adjudication for boundary ambiguity, true mislabelling ≈ 0.33% vs. 0.59% for NameRXN
Cross-taxonomy AMI vs. NameRXN: flat at 0.837 (L3) → 0.849 (L4) → 0.842 (L5), indicating matched resolution without spurious structure
Cross-taxonomy AMI vs. Rxn-INSIGHT: declining 0.724 → 0.720 → 0.691, reflecting Rxn-INSIGHT's lower resolution rather than LLM artifacts
Coverage on USPTO: Hybrid Classifier 98.6%, NameRXN 87.7%, Rxn-INSIGHT 55.3%
Coverage on CRD-2025: Hybrid Classifier 89.3%, NameRXN 68.3%, Rxn-INSIGHT 64.8%
Coverage on Ringbreaker: Hybrid Classifier 95.3%, NameRXN 77.4%, Rxn-INSIGHT 73.7%
Median latency: 3.2 ms/reaction (Hybrid Classifier) vs. 2.2 ms (NameRXN), 15.3 ms (Rxn-INSIGHT)
Taxonomy scaled to 14,073 classes; 179,495 reactions directly classified, extrapolated via template identity to 665,901

Limitations¶

LLM taxonomy estimated noise rate (2.19%) is ~3.7× higher than NameRXN (0.59%) before adjudication, even if comparable after boundary correction
Taxonomy extrapolation via template identity assumes all reactions sharing a template are mechanistically equivalent, which may not hold for diverse substrates
LLM-as-judge adjudication of boundary ambiguity is circular (same model family as pipeline); acknowledged but not fully resolved
SMIRKS generation loop not evaluated on truly out-of-distribution chemistry beyond the USPTO/CRD/Ringbreaker corpora shown
Paper is truncated; quantitative SMIRKS precision/recall tables and full coverage statistics for the SMIRKS-only classifier are only partially visible

Relevance to Harnesses / Meta-Harnesses¶

This paper is a direct instance of a multi-agent meta-harness in which the orchestration layer—not any single model—is the core contribution: five specialized agents with distinct roles (classify, verify, generate, aggregate, update) are composed into a feedback-driven pipeline that produces and validates symbolic artifacts (SMIRKS rules) at scale. The verification loop pattern—generate → corpus-test → error-feedback → refine → repeat—is a reusable harness motif applicable beyond chemistry wherever LLMs must produce formal, machine-checkable outputs. The generator-aggregator architecture for dynamic ontology extension is particularly relevant as a design pattern for self-expanding knowledge bases driven by agentic loops, generalizing to any domain where a taxonomy must adapt to a long-tailed data distribution without human intervention.