Zephyrus: An Agentic Framework for Weather Science

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Agentic framework integrating weather foundation models with language-based reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Zephyrus is the first agentic framework for weather science, coupling LLMs with a Python-based execution environment (ZephyrusWorld) that exposes forecasting models, climate simulators, and climatology tools via code generation. It introduces ZephyrusBench, a 2230-sample benchmark spanning 49 meteorological tasks at three difficulty levels. Zephyrus agents outperform text-only baselines by up to 44 percentage points, though hard tasks remain unsolved even by frontier LLMs.

Problem

Weather foundation models (GraphCast, FourCastNet, Stormer, etc.) operate exclusively on structured numerical data and have no natural language interface. LLMs, conversely, cannot reason over high-dimensional spatiotemporal atmospheric datasets. No prior system unified meteorological data with natural language reasoning for interactive, general-purpose scientific workflows. Existing weather-language models (e.g., CLLMate, WeatherQA) address only narrow subtasks using restricted variable subsets.

Method

ZephyrusWorld is a sandboxed Python execution environment providing five tools via clean Pythonic APIs: 1. WeatherBench 2 Data Indexer — xarray interface to ERA5 reanalysis (1979–2022, 1.5° / 6-hourly). 2. Geolocator — forward/reverse geocoding using Natural Earth; precomputed spatial caches via geopandas/shapely. 3. Forecaster — Stormer transformer model for short-to-medium range neural weather prediction. 4. Simulator — JCM intermediate-complexity atmospheric model on NeuralGCM's dynamical core (JAX, T32 ~3.5° resolution, 8 vertical layers; 5-day simulation in ≈25s on A100). 5. Climatology — precomputed ERA5 statistics (mean, median, extrema, std, quantiles) over 1979–2000 at all-time / seasonal / monthly / daily / 6-hourly timescales.

A FastAPI-based code execution server manages resource pools with acquire/release semantics, enabling true parallelism across tool instances without deadlocks.

Two agent variants are defined: - Zephyrus-Direct: generates a complete Python solution in one pass; runs an error-correction loop up to 20 times on execution failures. - Zephyrus-Reflective: ReAct-style multi-turn loop (max 20 turns) — generates code, observes execution output, analyzes scientific plausibility, then refines. Enables mid-trajectory correction of logical errors.

ZephyrusBench construction: 19 human-authored task types (human expert + graduate student) and 30 semi-synthetic types. Semi-synthetic pipeline uses a claim-extraction LLM agent (GPT-4.1) on NOAA reports to identify quantifiable claims → template → LLM-generated verification code → human review → ERA5 validation. Total: 2230 QA pairs across 49 tasks (Easy/Medium/Hard).

Evaluation uses task-type-specific metrics: Standardized Median Absolute Error (numerical), Median Absolute Error in hours (temporal), F1 + correctness (boolean), Earth Mover's Distance over geodesic grids (spatial), and NLI-based precision/recall (descriptive).

Key Contributions

• ZephyrusWorld: unified agentic environment exposing forecasting, simulation, climatology, and geospatial tools through a Python API with a parallel FastAPI execution server.
• Zephyrus-Direct: single-shot code-generation agent with error-correction loop.
• Zephyrus-Reflective: iterative ReAct-style agent with observation-driven refinement.
• ZephyrusBench: 2230 QA pairs across 49 meteorological task types (Easy/Medium/Hard), combining human-curated and semi-synthetically generated samples with scientifically verified ground truth.
• Task-type-specific evaluation metrics appropriate to meteorological answer types (numerical, temporal, boolean, spatial, descriptive).

Results

• Zephyrus agents outperform text-only (no-tool) baselines by up to 44 percentage points in correctness.
• Hard tasks remain difficult even for frontier LLMs, indicating ZephyrusBench presents a genuine unsolved challenge.
• (Full per-task ablation tables were not included in the provided excerpt.)

Limitations

• Hard tasks are not solved by current frontier models, suggesting the tool-use framework alone is insufficient for complex multi-step meteorological reasoning.
• The Stormer forecaster and JCM simulator have fixed resolution and scope; the environment does not cover ensemble forecasting or data assimilation pipelines.
• Semi-synthetic tasks are derived from NOAA reports and may not fully capture diversity of real operational workflows.
• Evaluation of descriptive answers relies on LLM-based NLI scoring, which may introduce its own errors.
• Text-only baselines are the primary comparison; no ablation against retrieval-augmented or fine-tuned meteorological LLMs is described in the excerpt.

Relevance to Agentic AI / LLM Agents

Zephyrus is a clean instantiation of the tool-augmented code-as-action agent paradigm in a high-stakes scientific domain, directly implementing and comparing Direct vs. ReAct-style reflective execution loops. It demonstrates that domain-specific tool environments — rather than end-to-end multimodal models — are currently the more effective path for grounding LLMs in structured numerical data. The semi-synthetic benchmark construction pipeline (claim-extraction agent → template → LLM-verified code) is a reusable methodology for building scientifically rigorous agentic benchmarks in other domains. The 44pp gap between tool-augmented agents and text-only baselines quantifies the core value proposition of agentic tool use in expert scientific contexts.