Skip to content

Reasoning With a Star: A Heliophysics Dataset and Benchmark for Agentic Scientific Reasoning

🕒 Published (v1): 2025-11-23 18:13 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RWS (Reasoning With a Star) is a 158-item heliophysics benchmark derived from NASA/UCAR Living With a Star problem sets, covering numeric, symbolic, and textual answer types. The paper also benchmarks four multi-agent patterns—HMAW, PACE, PHASE, and SCHEMA—against single-shot baselines, finding that no single pattern universally dominates and that task structure determines which coordination strategy is optimal.

Problem

Heliophysics is underrepresented in LLM reasoning benchmarks; existing datasets (GSM8K, MATH, GPQA) do not capture the domain-specific demands of physics reasoning—unit consistency, physical assumption tracking, LaTeX symbolic derivations, and structured output formats. Single-shot LLM prompting is unreliable for multi-step scientific reasoning requiring deductive inference rather than fact recall.

Method

Dataset construction: NASA/UCAR LWS Summer School problem sets were OCR-converted, manually cleaned, and normalized into JSONL with fields: problem statement, preamble, intermediate reasoning steps, ground-truth final answer, type label (numeric/symbolic/textual), format hints, and metadata (158 QA pairs).

Grader: Programmatic scoring via (i) unit-aware numerical tolerance (5%) for numeric answers, (ii) SymPy CAS for symbolic algebraic equivalence, (iii) semantic equivalence for textual answers. A two-agent LLM verifier (Parser + Judge, Gemini 2.5 Pro, temperature=1.0) handles edge cases where automatic matching fails.

Multi-agent patterns evaluated (all implemented via Google ADK): - HMAW: CEO→Manager→Worker hierarchical handoff, no retries - PACE: Plan→Answer→Critique→Enclose with one bounded retry on rejection - PHASE: Plan→Hypothesize→Analyze→Solve→Evaluate→Finalize with one bounded correction - SCHEMA: Architect→Allocator→Experts (0–3)→Synthesizer→Guard→Finalizer, dynamically allocating expert roles per task; inspired by MBSE/MAS-ZERO

The overarching design principle is STAR (Systems-engineering-of-Thoughts Agentic Reasoning): well-defined role modules, constrained interfaces, explicit guard conditions, and verification checkpoints mirroring NASA/INCOSE systems engineering handbooks.

Key Contributions

  • RWS dataset: 158 heliophysics QA pairs with reasoning steps, typed ground truth, and format hints; publicly available on HuggingFace (SpaceML/ReasoningWithAStar)
  • Programmatic grader supporting unit-aware numeric tolerance, CAS symbolic equivalence, and LLM-judge semantic verification
  • Comparative study of four multi-agent coordination patterns across six benchmarks (GSM8K, MATH, GPQA, RWS, HumanEval, SWE-bench Verified) under matched conditions and without domain-specific RAG
  • STAR design framework: a systems-engineering-inspired template for modular, interface-specified agentic architectures

Results

Single-shot accuracy on RWS (best to worst): - Gemini 2.5 Pro: 35.44% - OpenAI OSS 20B / OSS 120B: 32.91% (tied) - Meta Llama 3.3: 31.01% - Mistral 24.11: 27.22%

Multi-agent accuracy on RWS under Gemini 2.5 Pro: - SCHEMA: 44.31% (best on RWS) - PHASE: 42.51% - PACE: 41.92% - HMAW: 39.52% - All multi-agent patterns outperform the single-shot baseline (35.44%)

Cross-benchmark highlights (Gemini 2.5 Pro): - GSM8K: PACE best (93.41%), SCHEMA lowest (86.36%) - MATH: PACE best (81.51%), SCHEMA lowest (71.39%) - GPQA: HMAW best (79.01%), SCHEMA lowest (73.36%) - HumanEval: SCHEMA best (43.29%) - SWE-bench Verified: SCHEMA best (63.23%) - Macro-mean across all six datasets: PACE (64.57%) > PHASE (64.40%) > SCHEMA (63.66%) > HMAW (62.03%)

Limitations

  • Only 158 questions; coverage is limited to LWS Summer School problem sets from a subset of heliophysics topics
  • No RAG evaluated; domain-specific retrieval could substantially alter results
  • Textual answer grading relies on LLM judges (Gemini 2.5 Pro), introducing potential bias and inconsistency
  • Single base model (Gemini 2.5 Pro) used for multi-agent comparisons; results may not generalize across model families
  • SCHEMA and PHASE add coordination overhead without macro-mean improvements over PACE, suggesting cost–accuracy tradeoffs are not analyzed
  • No ablation of individual STAR components (e.g., guard agent, assumption tracking) in isolation

Relevance to Agentic AI / LLM Agents

This paper directly addresses multi-agent workflow design for scientific reasoning, providing empirical evidence that task structure—not agent count—determines which coordination pattern is optimal: self-critique loops (PACE) for arithmetic, structured role allocation with interface guards (SCHEMA) for constraint-heavy domains like physics and code. The STAR framework formalizes a systems-engineering approach to agentic architecture design (explicit role contracts, guard conditions, bounded retries) that is broadly applicable beyond heliophysics. The benchmark also isolates reasoning from retrieval, providing a clean signal for evaluating inherent multi-agent reasoning capability. For researchers tracking agentic AI, RWS is a new out-of-distribution domain benchmark where current SOTA tops out at ~44%, leaving substantial headroom and making it a useful evaluation target for future agent frameworks.