Skip to content

SciNav: A General Agent Framework for Scientific Coding Tasks

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

General agent framework for scientific coding; reasoning-to-execution pipeline [ICLR 2026]

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SciNav is an autonomous agent framework for scientific coding tasks that replaces brittle absolute scoring with pairwise relative judgments inside a structured tree search. It operates under constrained compute budgets without requiring pre-specified success metrics, making it practical for diverse scientific domains. On ScienceAgentBench and DA-Code, it consistently outperforms prior agent baselines across multiple LLM backbones.

Problem

Existing science agents target open-ended, full-cycle research (hypothesis generation, experiment design, report writing) whose outputs resist objective evaluation. Meanwhile, scientific coding benchmarks with executable, assessable outputs are approached with general-purpose agents (OpenHands, Auto-GPT) or engineering-centric pipelines that lack structured framework design for solution exploration. Specialized ML-coding frameworks like AIDE assume (i) well-defined task metrics available at runtime and (ii) large exploration budgets (e.g., 24-hour runs), neither of which holds for diverse, real-world scientific coding scenarios.

Method

SciNav implements Top-K Comparative Tree Search (TKCTS), a beam-search-style tree traversal governed by pairwise LLM judgments rather than pointwise scores. The algorithm maintains a priority queue of candidate solutions and at each iteration: (1) selects pairs for comparison via SelectPairs, (2) has an LLM determine which solution is better and why via Compare, (3) updates rankings, (4) prunes all but the Top-\(K\) branches (TopK), and (5) expands retained branches by generating child solutions (Expand). The process continues until a comparison budget \(B\) is exhausted, then selects the final solution via SelectFinal.

Four interacting components implement this loop: - Initial Planning: the LLM generates \(N\) diverse high-level plans (each visible to avoid repetition), then one executable program per plan, exploiting log-linear pass@K scaling. - Self-Debug: a code interpreter detects runtime errors; the LLM repairs them without discarding the full trajectory (up to 3 debug steps). - Self-Improve: iterative reflective refinement prompts the LLM to identify and act on one specific improvement per selected frontier solution. - Frontier Comparator: pairwise comparisons rank the frontier; Top-\(K\) candidates advance, the rest are pruned. Backtracking is implicit: up to \(K\) previously lower-ranked candidates can be revisited if the leading branch stagnates.

Hyperparameters: initial pool = 5 solutions, max debug steps = 3, total exploration steps = 10, temperature = 0.5 for generation/debug, 0 for comparison.

Key Contributions

  • Introduction of TKCTS, a budget-constrained tree search that uses relative pairwise judgments as the primary selection signal, eliminating the need for task-specific success metrics at runtime.
  • A structured, modular science agent framework combining planning, self-debug, self-improve, and a comparative frontier selector, applicable to diverse scientific coding task types and LLM backends.
  • Empirical demonstration across ScienceAgentBench and DA-Code using GPT-4o, Claude-3.7, and DeepSeek-R1, with ablation of comparator strategies.

Results

  • ScienceAgentBench (GPT-4o 2024-11-20): SciNav SR = 18.6% vs. Self-Debug SR = 15.0% — 24% relative gain in SR; VER 69.9% vs. 67.0% (+2.9 pp). API cost $0.342 vs. OpenHands $1.093.
  • ScienceAgentBench (GPT-4o 2024-05-13): SciNav SR = 16.1%, OpenHands = 13.1%, Self-Debug = 14.7%, Direct Prompting = 7.5%; SciNav beats OpenHands by 22.9% relative gain in SR.
  • Claude-3.7: SciNav SR = 25.5% vs. Self-Debug = 22.5% (+3 pp SR, −11.8 pp VER due to fewer debug steps).
  • DeepSeek-R1: SciNav SR = 19.6% vs. Self-Debug = 18.6% (+1 pp SR, +7.8 pp VER).
  • DA-Code (GPT-4o 2024-11-20): +29 pp absolute on data manipulation, +29 pp on statistical analysis, +13 pp average; gains of 12–23 pp across easy/medium/hard difficulties.
  • Frontier comparator ablation: Relative Judgments SR = 18.6% > LLM-Absolute = 16.2% > Random = 15.2%; Rubric-Absolute (with ground truth) = 21.1% (oracle upper bound).

Limitations

  • API cost is substantially higher than Self-Debug ($0.342 vs. $0.030 for GPT-4o 2024-11-20), limiting practical scalability to tasks where quality gains justify cost.
  • SciNav's VER can fall below Self-Debug when Self-Debug is allotted 10 debug steps vs. SciNav's 3 — the framework trades execution reliability for broader exploration coverage.
  • Gains over Self-Debug are smaller on DeepSeek-R1 (+1 pp SR), suggesting the benefit of relative-judgment guidance may diminish for models with strong intrinsic reasoning.
  • Rubric-Absolute with ground-truth outperforms all metric-free comparators, indicating a performance ceiling if reliable supervision were available — the metric-free design sacrifices some ceiling performance for generality.
  • Evaluated on two benchmarks; generalization to other scientific coding domains (e.g., physics simulation, symbolic computation) is unverified.

Relevance to Harnesses / Meta-Harnesses

SciNav is itself a meta-harness: it orchestrates multiple heterogeneous LLM invocations (planner, coder, debugger, comparator, refiner) under a shared budget through a single algorithmic backbone (TKCTS), exactly the kind of structured multi-step LLM pipeline a harness designer would build. The Frontier Comparator is particularly relevant — it shows how to implement a metric-free evaluation sub-harness using pairwise LLM calls, bypassing the need for any ground-truth signal during search. The explicit budget parameter \(B\) and the Top-\(K\) pruning mechanism offer a reusable pattern for harnesses that must bound cost while maintaining solution quality. The ablation of comparator strategies (Random, Absolute, Rubric, Relative) provides empirical guidance for choosing evaluation sub-components when composing meta-harnesses for domains without well-defined metrics.