AutoSynthesis: An agentic system for automated meta-analysis

🕒 Published (v1): 2026-07-16 17:45 UTC · Source: Arxiv · link

Why this paper was selected

Feuerriegel (TU Munich); end-to-end agentic scientific synthesis; reproducible for medical evidence automation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoSynthesis is an end-to-end multi-agent system that automates the full meta-analysis pipeline — from natural-language research question through literature retrieval, screening, effect-size computation, and random-effects synthesis — producing PRISMA-compliant reports. Applied to the contested question of LLM persuasiveness, its pooled estimate deviated by only \(\Delta g = 0.123\) Hedges' \(g\) from a published human benchmark, within the accepted \(\pm 0.20\) tolerance. This demonstrates that agentic LLM orchestration can approximate expert-conducted quantitative evidence synthesis at scale.

Problem

Producing a high-quality meta-analysis is prohibitively expensive and slow (>1,000 person-hours, often >1 year). Prior LLM-assisted systems address only subsets of the workflow — retrieval, screening, or extraction — but none compute standardized effect sizes or perform complete statistical synthesis, leaving a gap between literature automation and quantitative evidence synthesis.

Method

AutoSynthesis orchestrates a pipeline of specialized LLM agents across four stages:

1. Planning & retrieval: A planning agent converts a natural-language research question into a structured review protocol (PICO-style criteria, moderators, search queries); an automatic search agent queries scientific databases and deduplicates by title.
2. Eligibility & study mapping: A paper-reading agent fetches full-text PDFs; an eligibility agent filters by predefined criteria; a research-question identification agent maps independent studies/RQs within each paper.
3. Statistical extraction & validation: A statistical-result extractor agent pulls numerical statistics; a checker agent verifies each extracted row against the source text; a relevance-filter agent retains only results pertinent to the target meta-analysis.
4. Analysis & reporting: An analysis agent computes Hedges' \(g\) and Cohen's \(d\) from whatever statistics are available, fits a random-effects model, estimates between-study heterogeneity (\(I^2\), \(\tau^2\), Cochran's \(Q\)), runs trim-and-fill and funnel-plot publication-bias diagnostics, applies RoB 2 / ROBINS-I risk-of-bias rubrics, and a report agent writes a PRISMA-compliant Markdown output with forest plots, funnel plots, and full audit traces.

Key Contributions

• First end-to-end agentic system spanning the complete meta-analysis workflow including standardized effect-size computation and random-effects pooling, not just retrieval/screening.
• Modular, auditable architecture with full agent traces and intermediate decisions logged for human inspection.
• Support for moderator/heterogeneity analysis and principled publication-bias diagnostics (trim-and-fill, funnel plots, Egger's test) as part of the automated workflow.
• Demonstration that automated pooled estimates fall within the established \(\pm 0.20\) Hedges' \(g\) human-reanalysis tolerance region.
• Conceptual extension of "living" systematic reviews to "living" meta-analyses via near-real-time automated re-synthesis.

Results

• Literature retrieval recall: 71.4% of benchmark-included studies recovered (5/7); corrected recall 85.7% after accounting for post-cutoff publications and inaccessible PDFs.
• Literature retrieval precision: 62.5% raw; corrected precision 87.5% after qualitative adjudication.
• Pooled effect agreement: AutoSynthesis estimated Hedges' \(g = 0.143\) (95% CI [0.059, 0.226]) vs. benchmark \(g = 0.020\) (\(\Delta g = 0.123\)); both CIs overlapped substantially; deviation is within the accepted \(\pm 0.20\) tolerance.
• Heterogeneity agreement: Both found high between-study heterogeneity (\(I^2 = 88.3\%\) automated vs. \(75.97\%\) human).
• Publication bias agreement: Both detected small-study effects via Egger's test (\(p = 0.006\) automated vs. \(p = 0.018\) human).
• Study-level effect size correlation: Pearson \(r = 0.69\) (two-sided \(p = 0.085\)), OLS slope \(\beta = 1.62\) across 7 matched studies.
• The system screened 28 retrieved records, retained 25 after abstract screening, identified 19 eligible by full text, and extracted quantitative evidence from 8 studies (20 effect sizes).

Limitations

• Study retrieval is the primary source of error; PDFs not publicly accessible are excluded, introducing systematic coverage gaps.
• Evaluated on a single benchmark meta-analysis (LLM persuasiveness), limiting generalizability claims.
• Eligibility-assessment errors occur when articles use extensive framing around a topic without directly measuring the target outcome (one observed false inclusion).
• Not a replacement for expert judgment on research question formulation, moderator selection, and domain-specific interpretation.
• Statistical extraction requires convertible statistics; 6 of 19 eligible studies were dropped because their reported statistics could not be converted to an effect size.
• No ablation study isolating which agent contributes most to error; error attribution is qualitative.

Relevance to Agentic AI / LLM Agents

AutoSynthesis exemplifies a complex, real-world multi-agent pipeline where specialized agents with distinct roles (planning, retrieval, extraction, validation, analysis, reporting) are chained to accomplish a task that previously required months of human expert effort — a canonical demonstration of agentic LLM capability beyond chat or single-step tool use. The system's emphasis on auditability (full agent traces, per-step inspection) and structured output validation (statistical checker agent cross-referencing extracted numbers against source text) speaks directly to the reliability and verification challenges central to deploying LLM agents in high-stakes scientific workflows. For researchers tracking agentic AI, this work benchmarks agentic pipelines against a rigorous external ground truth (published expert meta-analyses) rather than synthetic tasks, providing a credible signal on where automation succeeds and where human oversight remains essential.