Skip to content

Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Earth-Agent is a ReAct-style agentic framework for Earth Observation (EO) that couples LLM reasoning with 104 specialized tools organized under the Model Context Protocol (MCP), enabling cross-modal, multi-step quantitative spatiotemporal analysis across RGB, spectral, and geodatabase product modalities. The authors also introduce Earth-Bench, a benchmark of 248 expert-curated tasks with 13,729 images and a dual-level evaluation protocol that scores both final answers and intermediate reasoning trajectories. Earth-Agent substantially outperforms both general-purpose agents and remote-sensing MLLMs on Earth-Bench.

Problem

Existing MLLM-based EO systems are restricted to RGB inputs, single-step/shallow reasoning, and cannot invoke domain-specific tools or scale to large image corpora. Prior EO agents address only perceptual tasks (change detection, scene classification), remain RGB-centric, lack a pre-defined structured tool ecosystem, and evaluate only final outputs—ignoring whether the reasoning trajectory is scientifically correct. Specifically, code-generation agents such as UnivEarth suffer high execution failure rates and do not perform genuine tool calling.

Method

Earth-Agent formalizes EO task solving as a Partially Observable MDP: \(\langle g, S, A, O, T \rangle\), where the LLM controller acts as policy \(\pi(a_t \mid g, m_t)\) over a history \(m_t = (o_0, a_0, \ldots, o_t)\). The ReAct loop iterates four steps: ① invoke a tool, ② append \((o_t, a_t, o_{t+1})\) to memory, ③ reason over updated memory, ④ select the next tool call, until a stopping condition is met.

The toolkit contains 104 tools grouped into five domain kits: - Index (12 tools): NDVI, NDWI, NBR, etc. - Inversion (18 tools): LST, PWV, vegetation water content, sea-ice concentration - Perception (10 tools): classification, detection, segmentation - Analysis (49 tools): trend detection, seasonality decomposition, change-point analysis, spatial autocorrelation - Statistics (15 tools): batch preprocessing, variance, skewness, cloud masking

All tools are exposed via MCP (Model Context Protocol), enabling extensibility and interoperability.

Earth-Bench construction: 248 questions are generated from GEE, NASA EarthData, and public RS datasets by a 12-person expert team. Each question is annotated at two levels: a Python solution (ordered function calls) yields ground-truth intermediate I/O per tool and a final answer, translated into structured JSON trajectory records. Two query regimes are supported—Auto-Planning (step-implicit, agent must autonomously plan the workflow) and Instruction-Following (step-explicit, agent follows human-provided step guidance).

Dual-level evaluation: - End-to-end: Accuracy (final answer) and Efficiency (trajectory length vs. expert). - Step-by-step: Tool-Any-Order (all required tools present), Tool-In-Order (correct partial order), Tool-Exact-Match (exact prefix match with expert trajectory), Parameter Accuracy (correct tool name + arguments per step).

Key Contributions

  • Earth-Agent: first agentic EO framework unifying RGB and spectral modalities via MCP-based 104-tool ecosystem and ReAct-POMDP control loop; easily extensible.
  • Earth-Bench: 248 expert-curated questions, 13,729 images, covering 14 task types (classification, detection, temperature monitoring, weather forecasting, etc.) across RGB/Spectrum/Products; average 5.4 reasoning steps per question.
  • Dual-level evaluation protocol: step-by-step trajectory scoring (4 metrics) + end-to-end scoring—enabling fine-grained diagnostics beyond final-answer accuracy.
  • Two query regimes (Auto-Planning and Instruction-Following) on the same 248 questions for orthogonal capability assessment.
  • Demonstrated superiority over general agents (Operator, Manus) and remote-sensing MLLMs on Earth-Bench and standard RS benchmarks.

Results

  • Earth-Agent substantially outperforms general-purpose agents Operator (OpenAI) and Manus on Earth-specific tasks in Earth-Bench.
  • Earth-Agent surpasses remote sensing MLLMs on standard remote sensing benchmarks.
  • Even SOTA LLM backbones (GPT-5, Gemini-2.5, Kimi-k2, DeepSeek-V3.1, Qwen3-max) operating within Earth-Agent achieve limited performance on Earth-Bench, confirming benchmark difficulty; scores across tasks cluster in the 30–60 range (per Figure 4, exact per-model numbers not fully reported in the provided text).
  • Earth-Bench requires an average of 5.4 reasoning steps per question (maximum 60+), far exceeding prior EO benchmarks (all at ≤1 step).

Limitations

  • Benchmark is small (248 questions), though image count is large (13,729); statistical coverage of the 14 task types may be thin per category.
  • The paper text is truncated and does not provide a dedicated limitations section; limitations below are partly evident from the text.
  • Performance of all tested LLMs "remains limited" even with the full toolkit—indicating the tool ecosystem may still be insufficient or misconfigured for the hardest tasks.
  • Tool coverage (104 tools) may not span all EO scientific domains; extending to new domains requires manual tool engineering.
  • UnivEarth's code-generation approach shows high execution failure rates—Earth-Agent's 104 tools sidestep this but create a fixed API surface that may not generalize to novel analytical needs without curation.
  • Dual-level evaluation requires annotating full reasoning trajectories, making Earth-Bench expensive to extend with new tasks.

Relevance to Harnesses / Meta-Harnesses

Earth-Agent is a concrete instantiation of a domain-specific harness: a structured ReAct control loop that orchestrates an LLM over a pre-defined, MCP-registered tool ecosystem, with explicit memory management and a defined stopping protocol. The five-kit taxonomy (Index → Inversion → Perception → Analysis → Statistics) represents a layered capability harness pattern analogous to how meta-harnesses decompose pipelines into stages. The dual-level evaluation protocol—scoring both trajectory quality and final output—is directly relevant to harness designers who need to verify not just that a pipeline produces the right answer, but that it executes the right sequence of steps, mirroring harness-level observability requirements. The use of MCP as an interoperability layer for tool registration is a reusable harness infrastructure pattern worth tracking for anyone building general-purpose orchestration frameworks.