Skip to content

Re4: Scientific Computing Agent with Rewriting, Resolution, Review and Revision

🕒 Published (v1): 2025-08-28 12:50 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Re4 is a multi-LLM agent framework for scientific computing that chains three specialized modules—Consultant, Programmer, and Reviewer—through a "rewriting-resolution-review-revision" loop orchestrated via LangGraph. The Reviewer module performs runtime-feedback-driven self-debugging and self-refinement without additional training, substantially lifting execution success rates across three Programmers tested.

Problem

Single reasoning LLMs generating scientific computing code suffer from low bug-free rates, random numerical method selection, hallucinated algorithms, and inability to solve challenging ill-conditioned problems (e.g., DeepSeek R1 achieving only 59% execution success on PDE benchmarks). No prior framework combined structured domain-knowledge augmentation, runtime-feedback review, and multi-model collaboration in a single end-to-end pipeline.

Method

The framework is orchestrated as a LangGraph state machine with three functional nodes communicating via Pydantic-validated JSON structured outputs:

  1. Consultant — rewrites vague natural language problem descriptions into structured mathematical context, identifies equation type (elliptic/parabolic/hyperbolic), and proposes algorithmic strategies (pseudocode, method alternatives).
  2. Programmer — generates modular Python code from the Consultant's augmented prompt; during revision loops, ingests error traces and Reviewer feedback instead.
  3. Reviewer — an independent LLM (always GPT-4.1-mini) that receives the Consultant context, generated code, and truncated runtime stdout/stderr (first+last 800 characters to fit context), then outputs: (a) algorithm suitability verdict, (b) bug diagnosis, (c) optimization suggestions. Forms a closed feedback loop with the Programmer for up to two revision rounds (rev-1, rev-2).

Multi-model deployment: GPT-4.1-mini, Gemini-2.5-flash-preview, and DeepSeek-R1 are used as Programmers; GPT-4.1-mini is fixed as Consultant and Reviewer for its 1M-token context window.

Key Contributions

  • Novel "rewriting-resolution-review-revision" logical chain for autonomous scientific computing code generation.
  • Independent Reviewer module that interfaces with code runtime outputs (stdout, warnings, errors) to drive self-debugging without extra training.
  • Multi-model collaborative architecture where distinct LLMs occupy distinct functional roles, outperforming any single model.
  • Evaluation across three heterogeneous scientific computing tasks: PDE solving (6 equations), ill-conditioned Hilbert linear systems (\(n \in \{5,10,\ldots,25\}\)), and data-driven dimensional analysis.

Results

Execution success rate (bug-free code + non-NaN solutions): - DeepSeek R1: 59% → 82% (+23 pp) - ChatGPT 4.1-mini: 66% → 87% (+21 pp) - Gemini-2.5-flash: 60% → 84% (+24 pp)

PDE benchmark (L2 relative error, representative improvements after rev-2 vs ans-0): - Lid-Driven Cavity: GPT-4.1-mini 4.2e-2 → 5.7e-2 (Gemini best at 9.9e-2 from 2.7e-1); DeepSeek R1 3.0e-1 → 1.8e-1 - Unsteady NS: DeepSeek R1 2.4e-1 → 2.0e-2 (~12× improvement) - Reviewer monotonically reduces median error and tightens IQR across all six PDE cases

Hilbert ill-conditioned systems (solving success rate, errors below \(L_\infty = 10^{-2}\)): - GPT-4.1-mini: 0% → 57% - Gemini-2.5-flash: 32% → 43% - DeepSeek R1: 11% → 46%

Data-driven dimensional analysis (keyhole dynamics): all three Programmers successfully identify dominant dimensionless quantities at rev-2; baseline success rates not numerically reported but shown improved in Figure 5.

Limitations

  • Only two Reviewer revision rounds (rev-1, rev-2) evaluated; convergence behavior for deeper iteration chains is not studied.
  • Reviewer and Consultant are fixed to GPT-4.1-mini; no ablation on Reviewer model choice or cross-model Reviewer configurations.
  • Runtime log truncation (800 characters each end) may discard critical intermediate error context in long execution traces.
  • Benchmark scope is narrow: six PDEs, one linear algebra problem class, one dimensional analysis case—generalization to broader scientific computing domains is asserted but not validated.
  • No wall-clock time or cost analysis relative to single-model baselines.
  • Randomness across 8 independent runs aggregated without confidence intervals reported in main text.

Relevance to Harnesses / Meta-Harnesses

Re4 is a concrete, domain-specialized meta-harness: it wraps heterogeneous LLMs in typed functional roles with structured I/O contracts (Pydantic JSON schemas) and wires them through a conditional-edge state machine (LangGraph), which is precisely the pattern of orchestration harnesses that compose and govern multiple agent components. The runtime-feedback loop—where code execution outputs are parsed and routed back as conditioning context—is a harness-level concern, not a model-level one, demonstrating how meta-harness design choices (truncation strategy, node isolation, context preservation) directly determine downstream model reliability. For researchers building harnesses, the paper shows quantitatively that the orchestration layer alone (no fine-tuning) yields 21–24 pp gains in execution success, reinforcing that harness architecture is often the dominant variable over model choice.