Paper Circle: An Open-source Multi-agent Research Discovery and Analysis Framework¶

🕒 Published (v1): 2026-04-07 17:59 UTC · Source: Arxiv · Venue: ACL · link

Why this paper was selected

Open-source multi-agent research discovery and analysis pipeline framework [ACL]

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Paper Circle is an open-source multi-agent framework for academic literature discovery and analysis, comprising a Discovery Pipeline (multi-source retrieval, scoring, ranking) and an Analysis Pipeline (typed knowledge graph construction with graph-aware Q&A). It is designed as a human-AI collaborative workbench rather than a fully autonomous system, emphasizing reproducibility via deterministic, structured artifacts at every agent step. Built on the smolagents CodeAgent/ToolCallingAgent substrate, it benchmarks favorably against BM25 and semantic retrieval baselines.

Problem¶

The exponential growth of scientific literature has outpaced existing discovery tools: single-source engines (arXiv, Semantic Scholar, Google Scholar) each miss 37–81% of a multi-source paper corpus, offer no bulk export with provenance logs, and lack deep structured understanding needed for rigorous literature reviews. Existing autonomous AI-Scientist systems prioritize full automation over human-in-the-loop curation and reproducibility.

Method¶

The system has two coordinated pipelines built on a smolagents CodeAgent (CoA) orchestrator with ToolCallingAgent (ToCA) workers:

Discovery Pipeline: An Intent Classification Agent parses queries into structured filters (mode, conference, year range). A Paper Search Agent performs offline/online retrieval across 8.7 sources on average. A Sorting Agent reranks via BM25, semantic similarity (all-MiniLM-L6-v2), recency, citations, or a cross-encoder (Qwen3-Reranker-0.6B). An Analysis Agent computes aggregate statistics; an Export Agent produces synchronized JSON/CSV/BibTeX/Markdown/HTML outputs. The orchestrator maintains a shared, persistent state updated iteratively via noising/denoising steps (inspired by TTD-DR).

Analysis Pipeline: PyMuPDF-based ingestion extracts structured elements (sections, figures, tables, equations). A SemanticChunker groups paragraph-level chunks up to 1500 characters while preserving figure/table/equation boundaries. Four CoA-based extractor agents (Concept, Method, Experiment, Linkage) build a typed MindGraph with provenance metadata (chunk IDs, page numbers, confidence scores). Graph-aware Q&A combines vector retrieval (EmbeddingStore) with 1-hop graph expansion via GraphRetriever. A CoverageChecker flags unlinked elements for quality assurance.

A seven-agent Review Framework (Deep Analyzer, Critic, Literature Expert, Contribution Analyzer, Reproducibility Checker, Summarizer, Knowledge Graph Agent) runs in parallel via ThreadPoolExecutor.

Key Contributions¶

Open-source multi-agent research discovery and analysis system covering the full literature engagement lifecycle
Deterministic, step-logged pipeline producing synchronized structured exports (JSON, BibTeX, Markdown, CSV, HTML) with full provenance
Typed knowledge graph schema with node/edge provenance (source chunk IDs, page numbers, verification status, confidence scores)
Built-in evaluation harness computing MRR, Recall@K, Precision@K, and hit rate per agent step, stored for longitudinal tracking
Multi-agent peer-review framework with parallel specialized roles (critic, reproducibility checker, contribution analyzer)
Coverage verification layer preventing silent omissions in knowledge graph construction

Results¶

Best retrieval: Qwen3-Coder-30B-Q3_K_M achieves 80% hit rate, MRR = 0.627, R@10 = 0.74 on 50-query SemanticBench
BM25 baseline is highly competitive at 78% hit rate, MRR = 0.541, suggesting lexical matching remains strong
On 500-query RA-Bench (LLM-generated queries), the same Qwen3-Coder model achieves 98% hit rate, MRR = 0.882, R@1 = 0.83
Multi-source coverage: PaperCircle queries 8.7 sources on average; only 9% of its 21,115-paper corpus is unretrievable, vs. 70.9% (arXiv), 80.4% (Semantic Scholar), 36.9% (Google Scholar) from single sources
NASA-TLX usability score: 1.2/7 overall workload; positive UX items average 7.6/10; evaluated across 81 real-world discovery sessions spanning 9 domains
Review quality: chat-style LLMs (e.g., gpt-oss) outperform code-oriented models for coherent end-to-end review; quality improves with scale

Limitations¶

Benchmark corpus is small (50 queries for SemanticBench) with 292 total conference papers, limiting statistical confidence
PDF availability is only 62.5% across the multi-source corpus, meaning many papers cannot be fully analyzed
RA-Bench queries are synthetically generated by a 20B LLM from paper metadata, potentially inflating retrieval scores
Review evaluation relies on ICLR 2024 reviews as ground truth over only 50 papers; no inter-annotator agreement reported
All experiments run on a fixed 4×40 GB GPU setup with Ollama; performance under different hardware/quantization budgets not systematically studied
Reproducibility Checker relies on self-reported metadata rather than actual code execution or artifact verification

Relevance to Harnesses / Meta-Harnesses¶

Paper Circle is a concrete implementation of a multi-agent meta-harness pattern: a CodeAgent orchestrator dispatches to specialized ToolCallingAgent workers, maintains shared persistent state across steps, and synchronizes structured outputs at every pipeline stage — precisely the architecture meta-harnesses must solve. Its deterministic, step-logged design with JSON state files mirrors best practices for reproducible harness construction, making it a reference for how to wire together discovery, analysis, and export subagents without silent state loss. The built-in evaluation harness (MRR/Recall@K per agent step, batch parallelism) is directly applicable to anyone designing self-monitoring or auto-benchmarking meta-harnesses. The Coverage Verification and CoverageChecker components demonstrate a lightweight quality-assurance layer that meta-harness designers can adapt to detect silent omissions in multi-agent pipelines.