Skip to content

AstroReview: An LLM-driven Multi-Agent Framework for Telescope Proposal Peer Review and Refinement

🕒 Published (v1): 2025-12-31 09:55 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AstroReview is a three-stage multi-agent LLM framework that automates peer review of telescope observing proposals, covering novelty assessment, feasibility simulation, and meta-review with reliability verification. Without domain fine-tuning, it achieves 87% accuracy identifying accepted HST proposals, and a closed-loop review-refinement cycle boosts proposal acceptance rates by 66% in two iterations.

Problem

Telescope Time Allocation Committees face unsustainable workloads—hundreds to thousands of proposals per cycle—leading to reviewer fatigue, inconsistent grading, and opaque feedback. Existing AI tools treat LLMs only as pre-processing aids for reviewer matching rather than as direct proposal evaluators or feedback providers. A prior ESO pilot using ChatGPT-3.5/4 directly found generic LLMs provided little benefit for scoring or revision.

Method

AstroReview chains three sequential agent stages: 1. Novelty & Scientific Merit: A semantic parser extracts targets and research goals; a question-generation module crafts evaluative queries; a retrieve-and-rank process searches literature and databases; a novelty assessor synthesizes a judgment. 2. Feasibility & Expected Yield: A parameter parser extracts instrument settings and orbital constraints; official mission simulation tools run visibility prediction and synthetic exposure simulations to produce SNR diagnostics. 3. Meta-Review & Reliability Verification: Multiple independent Review Agents (Qwen-2.5-72B, temperature 0.4) score proposals on six dimensions; a Meta-Review Agent consolidates scores and issues a final decision; a Reliability Verifier audits template compliance, evidence-claim alignment, and logical consistency, rewriting detected errors.

For iterative refinement ("AstroReview in Action"), a Proposal Authoring Agent revises drafts using reviewer feedback stored in a manually-injected memory buffer. A Decision Gate halts iteration when cosine similarity between consecutive drafts exceeds 0.90 and score improvement is <1 point, or after three rounds maximum.

Key Contributions

  • Three-stage task-isolated pipeline that localizes reasoning errors and suppresses hallucinations without domain fine-tuning.
  • Reliability Verification module: post-hoc audit that enforces structural compliance and corrects internal inconsistencies in generated reviews, shown to be the key driver of classification accuracy gains.
  • Closed-loop Proposal Authoring Agent + Review Agent cycle demonstrating iterative automated refinement.
  • Dual-signal Decision Gate (cosine similarity + score convergence) preventing premature or unnecessary loop continuation.
  • Ablation study showing the stack of multi-reviewer + meta-review + reliability verification outperforms single-reviewer and partial configurations on positive-sample accuracy.

Results

  • Decision accuracy (Stage 3 only, Qwen-2.5-72B, 26,822 HST abstracts, balanced 1:1 dataset):
  • Single Reviewer: 79.78% positive accuracy, 58.46% negative accuracy
  • Multi-Reviewer + Meta-Review: 80.51% positive / 48.16% negative
  • Full stack (+ Reliability Verification): 87.08% positive / 46.69% negative
  • Proposal improvement (3 refinement rounds, single reviewer):
  • Acceptance rate: ~33% (Round 0) → ~99% (Round 2), a 66% increase after two iterations
  • Mean score dropped 0.99 points in Round 1 (indicating corrections of fundamental weaknesses) then only 0.09 in Round 2
  • Score standard deviation contracted from 0.59 to 0.23 after Round 1
  • Negative-sample accuracy degrades with the full stack (46.69%, below random), attributed to synthetic negatives retaining attractive scientific content from their accepted-proposal origins.

Limitations

  • Experiments rely exclusively on HST proposal abstracts (positive samples only); no genuine rejected proposals available, requiring synthetic degradation that preserves scientific appeal and produces unreliable negative-class evaluation.
  • Stages 1 and 2 (novelty and feasibility) were bypassed in all reported experiments because abstract-only data lacks the structured parameters those stages require; full-proposal evaluation is untested.
  • Negative-sample accuracy falls below chance (46.69%) with the strongest configuration, making real-world rejection detection uncertain.
  • Only two Review Agents used in multi-reviewer stage; more agents produced redundant comments with diminishing diversity.
  • Evaluation rubric is intentionally domain-agnostic, differing from official observatory criteria, limiting direct comparability to TAC decisions.
  • No comparison against human reviewer baselines on the same dataset.

Relevance to Harnesses / Meta-Harnesses

AstroReview is a domain-specific harness instantiating the review-refine loop pattern central to meta-harness design: sequential specialized agents feed structured outputs forward, a meta-agent synthesizes cross-agent signals, and a verifier performs post-hoc correction before results are committed. The explicit Decision Gate with dual convergence signals (similarity + score delta) is a reusable termination mechanism for iterative agent loops. The manual memory injection strategy—bypassing native LLM memory to prevent context overflow in multi-round cycles—is a practical workaround applicable to any long-horizon harness. The ablation isolating the reliability verification stage's contribution to accuracy demonstrates how audit agents function as quality control layers in multi-stage pipelines.