Skip to content

EWE: An Agentic Framework for Extreme Weather Analysis

🕒 Published (v1): 2025-11-26 14:37 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EWE (Extreme Weather Expert) is the first agentic framework dedicated to automated diagnostic analysis of extreme weather events, emulating expert workflows through knowledge-guided planning, a dual-auditor closed-loop reasoning engine, and a domain-specific meteorological toolkit. It also introduces a 103-event benchmark with a step-wise, MLLM-as-judge evaluation metric covering seven analytical stages. The framework targets the gap between powerful AI forecasters (e.g., Pangu-Weather) and the still-manual, expert-bottlenecked post-event physical mechanism diagnosis.

Problem

AI for Earth science has prioritized prognostic models (GraphCast, FengWu, Pangu-Weather) that produce forecasts but cannot reason about why an event occurred. Post-event diagnostic analysis—reconstructing physical mechanisms such as potential vorticity advection, vorticity fields, and thermodynamic forcing—remains a manual, expert-intensive process that cannot scale to the growing volume of high-impact events. Existing LLM/agent applications in meteorology execute discrete human-prompted subtasks rather than orchestrating an autonomous end-to-end diagnostic inquiry. No benchmark existed for this task.

Method

EWE formalizes extreme-event diagnosis as an autonomous iterative trajectory \(\tau = (t_k, a_k, o_k, i_k)_{k=1}^{N}\), cycling through Thought, Action, Observation, and Interpretation, driven by a multimodal LLM backbone. Three integrated components:

  1. Knowledge-Enhanced Planning: Expert-annotated chain-of-thought exemplars are used as one-shot prompts to decompose the diagnostic goal into ordered, knowledge-anchored sub-goals (e.g., circulation identification → diagnostic field computation → mesoscale analysis). These CoT guidelines are also stored in the agent's memory module, constraining both high-level planning and low-level execution to domain physics.

  2. Self-Evolving Closed-Loop Reasoning: After each action (typically Python code for data processing or visualization), a dual-auditor checks the output:

  3. Code Auditor: static analysis for latent bugs—incorrect tool parameters, flawed data indexing—that bypass runtime exceptions but corrupt scientific conclusions.

  4. Content Auditor: semantic/perceptual quality of visualizations (over-plotting, low-contrast colormaps, occluded labels). Both auditors feed back into the next thought phase. A dynamic memory module distills completed sub-task findings into concise summaries and prunes transient context to maintain token tractability.

  5. Meteorological Toolkit: Wraps ERA5 reanalysis (0.25°, >300 TB) with a data-agnostic acquisition tool (NetCDF output with 30-year climatological baselines) and a curated library of expert-verified Python functions for canonical diagnostics (Integrated Vapor Transport, potential vorticity, etc.).

Benchmark: 103 events curated from EM-DAT and WMO reports, covering IPCC AR6 categories (temperature extremes, precipitation, drought, tropical/extratropical cyclones). Step-wise evaluation assigns scalar reward \(r_k = E(e, a_k, o_k, i_k)\) per step, classified into 7 stage types, evaluated by MLLM judge (GPT-4.1) under Single-Response Grading (SG) and Comparative Grading (CG) protocols with LLM-generated, human-verified checklists.

Key Contributions

  • First end-to-end agentic framework for automated extreme weather diagnostic analysis
  • Dual-auditor module (Code Auditor + Content Auditor) for self-correction beyond environment feedback
  • Benchmark of 103 high-impact events with global coverage across six IPCC AR6 categories
  • Step-wise, category-aware MLLM-as-judge evaluation across seven analytical stages
  • ERA5-backed meteorological toolkit with expert-verified diagnostic functions

Results

Scores normalized to [0, 1] across 7 stages; five frontier models evaluated (SG = Single-Response, CG = Comparative Grading):

  • Claude-4-Sonnet achieves highest CG Report score (0.950) and strong CG Identification (0.832) and Synoptic (0.837)
  • GPT-4.1 leads SG Plan (0.832) and SG Report (0.947); strong overall in SG
  • Gemini-2.5-Pro leads SG Report (0.898); competitive across stages
  • o4-mini competitive in Synoptic CG (0.816) and Thermo CG (0.720)
  • Llama-4-Maverick consistently lowest across nearly all stages and both protocols (e.g., SG Report 0.587, CG Report 0.486)
  • Step cap at 40 actions per run; tasks exceeding this marked as failures

Limitations

  • Evaluation relies on a single LLM judge (GPT-4.1); Gemini-2.5-Pro was dropped as judge due to systematic penalization of correct code, indicating evaluator brittleness
  • Benchmark limited to 103 events; distribution skewed geographically toward Asia (32%), Europe (24.3%), North America (21.4%)
  • Agent runs capped at 40 steps; no analysis of how often this cap is hit or what proportion of complex events require more
  • No ablation of dual-auditor components reported in the truncated text
  • Paper uses ERA5 reanalysis only; generalization to real-time observational streams is claimed but not demonstrated

Relevance to Harnesses / Meta-Harnesses

EWE is a domain-specific agentic harness: it wraps an MLLM with structured planning, tool orchestration, closed-loop execution, and dynamic memory management—exactly the pattern of a task harness that constrains model behavior via external scaffolding rather than relying on raw model capability. The dual-auditor module is a harness-level quality gate enforcing both code correctness and output quality before the agent advances, a pattern transferable to any harness requiring intermediate output validation. The step-wise benchmark with LLM-as-judge and category-aware checklists constitutes a meta-evaluation harness for agent trajectories—a methodology applicable to any multi-step agentic system where credit assignment across intermediate steps matters. The explicit separation of knowledge (CoT exemplars in memory), tools (meteorological toolkit), and reasoning loop (closed-loop MLLM) is a clean harness decomposition relevant to anyone designing domain-specific orchestration frameworks.