Skip to content

Deep Research in Physical Sciences: A Multi-Agent Framework and Comprehensive Benchmark

🕒 Published (v1): 2026-06-17 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Multi-agent deep research framework + comprehensive benchmark for physical sciences

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PhySciBench is a 200-question expert-curated benchmark covering six physical-science task categories across three cognitive stages, revealing that even the strongest generalist agent (Gemini Deep Research) achieves only 33.5% accuracy. DelveAgent, a modular multi-agent framework built around adaptive planning, dual-granularity memory, and hierarchical physics-grounded reflection, reaches 41.0% on PhySciBench (+7.5 pp) at roughly one-third the inference cost of the strongest baseline.

Problem

Existing benchmarks for AI in the physical sciences either test isolated single tasks (no extended tool-use workflows) or are text-only (missing multimodal parsing, structured data extraction, and code execution). No benchmark faithfully captures the end-to-end cognitive workflow of physical-science research, and no systematic failure analysis has identified what specifically limits current deep-research agents in this domain.

Method

PhySciBench construction: 200 questions authored by domain experts from recent high-impact publications and adapted textbooks, stratified into six categories across three cognitive stages — Information Extraction (Multimodal QA, Structured Information Extraction), Scientific Understanding (Long-context QA, Scientific Reasoning), and Application & Creation (Experimental Design, Code Generation). Questions are balanced 100/100 physics/chemistry; most require 20–30 reasoning steps and 2–4 tool classes. Evaluation uses a composite pipeline: exact-match normalization, rule-based syntax/key-value checks, rubric-guided LLM judge, and sandboxed code execution (Spearman \(\rho = 0.80\) vs. expert grading).

DelveAgent architecture: A central Planner orchestrates specialized Workers with three targeted modules: 1. Adaptive Planning Loop — iterates initial-plan → subtask execution → intermediate observation → plan refinement, enabling dynamic replanning rather than committed decomposition. 2. Dual-Granularity Memory — pairs an experience store (successful trajectories, recovery strategies, planning patterns) with a knowledge store (equations, procedures, physical constants); both stores are shared between Planner and Workers and updated by a Learner module after verified trajectories. 3. Hierarchical Physics-Grounded Reflection — a local verifier checking units, dimensions, formulas, code execution, and tool outputs at the step level, plus a global critic checking source grounding, scientific consistency, and format compliance at the trajectory level; triggers corrective re-execution before final response.

Key Contributions

  • PhySciBench: first benchmark explicitly targeting end-to-end physical-science research workflows with multimodal inputs, structured extraction, code execution, and multi-step tool use; 200 expert-curated questions with anti-contamination safeguards.
  • Composite evaluation pipeline with Spearman \(\rho = 0.80\) against expert grading, substantially above single-LLM judge (\(\rho = 0.57\)), ROUGE-2 (\(\rho = 0.31\)), and BERTScore-F1 (\(\rho = 0.15\)).
  • Failure taxonomy of 326 verified errors from Gemini Deep Research: planning/reasoning (46%), execution/response (31%), knowledge/implementation (23%).
  • DelveAgent framework demonstrating that architectural specialization — not model scale — accounts for the bulk of the improvement; ablations show each module contributes ~2.5–3.0 pp independently.

Results

  • DelveAgent achieves 41.0% overall on PhySciBench vs. 33.5% for Gemini Deep Research (+7.5 pp), 30.0% for OpenAI Deep Research, 21.5% for ODR-smolagents.
  • Category-level gains: Structured Information Extraction +15.0 pp (55.0% vs. 40.0%), Code Generation +15.0 pp (45.0% vs. 30.0%), Long-context QA +10.0 pp (23.33% vs. 13.33%), Multimodal QA +7.7 pp (43.58% vs. 35.90%).
  • No gain on Experimental Design (17.14% matches baseline) or Scientific Reasoning (58.33% matches Gemini Deep Research).
  • Ablations: removing Dual-Granularity Memory −3.0 pp; removing Hierarchical Reflection −3.0 pp; removing Adaptive Planning Loop −2.5 pp; removing all three simultaneously −6.5 pp (vs. sum 8.5 pp, indicating partial overlap).
  • Stripped-down DelveAgent narrows gap to Gemini Deep Research from 7.5 pp to 1.0 pp, confirming architectural components — not the base model — drive gains.
  • On three additional benchmarks: HLE 34.94% vs. 33.67% (GDR), SGI-DR +6.98 pp, FS-Research comparable; highest average across three (23.62% vs. 20.87%).
  • Inference cost approximately one-third of Gemini Deep Research.

Limitations

  • Absolute accuracy remains low (41.0%); benchmark is deliberately unsaturated, but even DelveAgent fails on ~59% of questions.
  • Experimental Design shows no improvement over the strongest baseline, suggesting the reflection and memory modules do not address the core difficulty of open-ended protocol generation.
  • Residual failure modes include evidence binding errors, constraint satisfaction failures, formal-specification fidelity issues, and cross-document disambiguation — the latter two reflect fundamental limits of current multimodal foundation models, not fixable by architecture alone.
  • Benchmark is concentrated in condensed-matter physics and synthetic organic chemistry/catalysis; coverage of other subfields is limited.
  • Benchmark size (200 questions) limits statistical resolution for per-category ablations.
  • Real-world case studies (Section 2.5) are described qualitatively; no quantitative scoring is reported in the excerpt.

Relevance to Harnesses / Meta-Harnesses

DelveAgent is a concrete instantiation of a meta-harness pattern: a modular orchestration layer that wraps tool-using subagents with adaptive planning, a dual-granularity memory store, and a hierarchical reflection loop, making it directly analogous to harness scaffolds like OpenDevin or AutoGen but evaluated in a rigorous, domain-specific benchmark setting. The 7.5 pp gain over Gemini Deep Research at one-third the cost is evidence that explicit scaffolding decisions — what to plan, what to remember, when to reflect — matter more than raw model capability, which is a key thesis for anyone building general agent harnesses. PhySciBench's three cognitive stages (recall, reasoning, synthesis) and six task categories also offer a blueprint for how to stress-test a harness beyond pass/fail coding benchmarks, which is directly applicable to evaluating harness robustness in medical or multimodal science contexts.