Skip to content

SciNav: A General Agent Framework for Scientific Coding Tasks

🕒 Published (v1): 2026-03-11 11:57 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SciNav is an autonomous agent framework for scientific coding tasks that uses Top-K Comparative Tree Search (TKCTS), replacing absolute LLM scoring with pairwise relative judgments to guide solution selection and pruning under constrained budgets. It outperforms prior agent baselines (OpenHands, Self-Debug) on ScienceAgentBench and DA-Code across multiple base models. The core insight is that comparative assessments yield sharper, more reliable quality signals than pointwise scoring when no ground-truth success metric is available at runtime.

Problem

Existing science agents target open-ended, full-cycle research tasks (hypothesis generation, report writing) whose outputs resist objective evaluation. Scientific coding benchmarks provide executable, objectively assessable tasks, but agent approaches to these benchmarks are ad-hoc engineering pipelines lacking structured framework design. Prior specialized agents (e.g., AIDE) assume (i) well-defined task metrics available during search and (ii) large exploration budgets (e.g., 24-hour runs), neither of which is practical when task types are diverse and metric definitions vary per problem.

Method

SciNav wraps an LLM in a Top-K Comparative Tree Search (TKCTS) loop:

  1. Initial planning and solution generation: The LLM generates \(K\) diverse high-level plans (each plan visible to the model before generating the next to avoid repetition), then produces one candidate program per plan.
  2. Self-debug: A code interpreter detects and repairs runtime errors within each candidate trajectory (up to 3 debug steps), enabling on-the-fly error correction without discarding trajectories.
  3. Frontier Comparator: Candidate pairs \((s_i, s_j)\) are submitted to an LLM at temperature 0 for a pairwise preference judgment. Rankings are updated via a ranking algorithm (Appendix F); the top-\(K\) candidates are retained and expanded, while low-potential branches are pruned. Budget is counted in pairwise comparisons.
  4. Iterative self-improve: Along retained branches, the model identifies a specific refinement point in the current frontier solution and rewrites it, mirroring iterative human revision.

The total exploration budget is fixed (initial solutions = 5, max debug steps = 3, total exploration steps = 10); no task-specific success metric is required during search.

Key Contributions

  • TKCTS: a structured tree search that uses pairwise relative judgments instead of absolute scoring to select and prune solution branches under a fixed comparison budget.
  • A general science agent framework that operates without pre-specified evaluation metrics, making it applicable to diverse scientific coding tasks where gold signals are unavailable at runtime.
  • Empirical ablation showing relative judgments outperform random selection, LLM-absolute scoring, and approach rubric-based absolute scoring (which requires ground-truth access).
  • Demonstrated generalization across four base models (GPT-4o-0513, GPT-4o-1120, Claude-3.7, DeepSeek-R1) and two benchmarks (ScienceAgentBench, DA-Code).

Results

  • ScienceAgentBench (GPT-4o-1120): SciNav SR = 18.6% vs. Self-Debug SR = 15.0% (+24% relative); VER = 69.9% vs. 67.0%; cost $0.342 vs. $0.030 per task.
  • ScienceAgentBench (GPT-4o-0513): SciNav SR = 16.1% vs. OpenHands = 13.1% (+22.9% relative), vs. Self-Debug = 14.7%; cost $0.512 vs. $1.093 (OpenHands) and $0.057 (Self-Debug).
  • ScienceAgentBench (Claude-3.7): SciNav SR = 25.5% vs. Self-Debug = 22.5% (+3 pts); VER = 72.5% vs. 84.3% (SciNav runs fewer debug steps).
  • DA-Code (GPT-4o-1120): +29 absolute points on data manipulation and statistical analysis; +13 absolute points average; +10%–23% across easy/medium/hard difficulty levels.
  • Frontier comparator ablation: Relative Judgments SR = 18.6% > LLM-Absolute = 16.2% > Random = 15.2%; Rubric-Absolute (with ground-truth access) = 21.1% serves as an oracle upper bound.

Limitations

  • Higher per-task API cost than simpler baselines (e.g., $0.342 vs. $0.030 for Self-Debug on GPT-4o-1120); the cost-quality trade-off may be prohibitive at scale.
  • Absolute success rates remain low (≤25.5% SR on ScienceAgentBench), indicating the benchmark is far from solved.
  • The frontier comparator requires an LLM call at temperature 0 per pair; the total comparison budget is a hard constraint that limits search depth.
  • Evaluation is limited to code-output tasks; the framework is not validated on open-ended scientific reasoning where outputs are non-executable.
  • Budget hyperparameters (initial solutions = 5, max debug = 3, total steps = 10) are fixed across all tasks and may not generalize optimally to tasks with different complexity profiles.

Relevance to Harnesses / Meta-Harnesses

SciNav is a concrete example of a meta-harness pattern: a structured orchestration layer that wraps a base LLM with multi-stage loops (plan generation → self-debug → comparative selection → self-improve) and a budget-aware scheduler, rather than calling the model in a single pass. The TKCTS algorithm functions as an inner harness loop with explicit state (priority queue, comparison budget counter, beam of \(K\) retained branches), making the orchestration logic separable from the underlying model. The Frontier Comparator component—using LLM-as-judge in pairwise mode to steer search—is directly applicable to any harness that needs model-free success metrics, a common constraint in harness design for open-domain tasks. The cost-tracking metric (\(\$/\text{task}\)) and ablation of selection strategies provide a practical template for evaluating harness design choices.