Bringing Agentic Search to Earth Observation Data Discovery¶

🕒 Published (v1): 2026-07-02 16:24 UTC · Source: Arxiv · link

Why this paper was selected

Deployed agentic search over NASA geoscience data; concrete real-world agentic retrieval system

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents a three-stage agentic search system for NASA Earth Observation dataset discovery, pairing a hybrid BM25 + fine-tuned neural retriever with an LLM reranker that autonomously invokes web search and arXiv lookups. They also release NASA-EO-Bench, a 47k query–dataset benchmark derived from citation graphs, enabling the first controlled same-model comparison of agentic vs. plain LLM reranking in geoscience retrieval.

Problem¶

Finding NASA datasets across thousands of entries in fragmented archives (CMR, DAACs) is hard even for domain experts. General-purpose LLMs suffer from domain shift on geoscience terminology, and standard RAG pipelines are bounded by context-window limits and embedding mean-bias, leaving ranking quality as the critical bottleneck. No prior geoscience retrieval benchmark was large enough to support supervised training or stable metric estimation (the previous best, ReSearch, had only 28 queries).

Method¶

The pipeline has three stages:

Stage 1 — Router: Dispatches queries directly to NASA official tools (Harmony, SDE, WorldView, Giovanni); queries fully resolved here terminate early.

Stage 2 — Hybrid retrieval: Combines BM25 (lexical anchor for exact instrument/mission tokens) with a task-adaptive neural scorer. Two variants are evaluated: (a) NN-SSC — a frozen encoder plus a lightweight MLP $f_\theta: \mathbb{R}^{2D} \to \mathbb{R}$ that takes concatenated query–dataset embeddings and outputs a pair-specific relevance scalar; (b) encoder fine-tuning on citation-grounded (query, positive, hard-negative) triples under a log-sum-exp ranking loss: $$\ell_p = \mathrm{logaddexp}(s_p,\; \mathrm{logsumexp}_{n \in \mathcal{N}}\, s_n) - s_p.$$ Scores are fused via a convex combination $s_\text{hybrid} = \alpha \cdot \hat{s}_\text{NN} + (1-\alpha)\cdot\hat{s}_\text{BM25}$, where $\alpha = \pi_n / (\pi_\ell + \pi_n)$ is set analytically from standalone training-split performance.

Stage 3 — Agentic reranking: The top-K candidates are passed to an LLM that executes a five-step research routine (web search + arXiv paper lookup) before producing the final ranking. This is contrasted against a single-shot LLM rerank prompt on the same model and candidate set, holding the output contract fixed. Evaluation uses Recall@K, MRR, MAP (citation-based), and LLM-as-a-Judge Precision@K (Qwen3.6-35B-A3B served via vLLM).

Key Contributions¶

NASA-EO-Bench: 47,654 citation-grounded query–dataset pairs (21,272 task-based queries) over 8,058 NASA CMR datasets; 38k/9k train/test split stratified by citation count; ~760× larger than prior geoscience retrieval benchmarks.
Retrieval suite: NN-SSC pairwise score correction, fine-tuned sentence transformer, and analytically-weighted BM25 fusion; lifts R@10 and MRR by >5× over the unadapted cosine baseline.
Controlled agentic-vs-LLM-rerank comparison: Same model (Opus 4.7 and DeepSeek v4 pro), same candidate set, same evaluation protocol — isolating the contribution of autonomous tool access from model choice.
Deployed public service for the geoscience community.

Results¶

Hybrid retrieval suite (BM25 + NN-SSC or fine-tuned encoder) lifts both R@10 and MRR by >5× over the unadapted cosine similarity baseline.
Agentic reranking (five-step web + arXiv routine with autonomous tool calls) yields a directional MAP/MRR gain over single-shot LLM reranking on the stratified $N=200$ test subset for both Opus 4.7 and DeepSeek v4 pro.
Agentic reranking lifts MRR by 28% on the $N=200$ stratified subset relative to the non-agentic LLM rerank baseline (zero-shot; no additional training).

Limitations¶

Citation-based silver labels are recall-biased: datasets relevant to a query but never cited by training papers are penalized; ground truth is use-based, not completeness-based.
Benchmark inherits NASA EO-KG coverage and curation bias: queries derive from GES DISC publications only; other DAACs are not represented.
Agentic evaluation is on a small stratified subset ($N=200$), limiting the statistical power of the MAP/MRR gain estimates.
Routine and tool access are coupled by design: the five-step research routine explicitly names specific tools, so the contribution of the routine structure vs. tool access per se cannot be independently ablated.
Matthew effect in citation data: popular datasets are over-represented as positives; rare but relevant datasets may be systematically under-retrieved.
No end-to-end evaluation: only the retrieve-and-rerank layer is studied; full pipeline QA accuracy is not measured.

Relevance to Agentic AI / LLM Agents¶

This paper offers a concrete, domain-specific instantiation of the agentic-reranking paradigm — where an LLM autonomously calls external tools (web search, arXiv) mid-task rather than relying solely on inlined context — and provides the first controlled same-model ablation distinguishing tool-augmented agents from plain LLM prompting in an IR setting. The finding that agentic tool use yields directional gains over a zero-shot LLM reranker (28% MRR lift) without any additional training supports the broader hypothesis that tool access is complementary to supervised retrieval, not redundant with it. The three-stage router → retriever → agentic-reranker architecture is a transferable template for agentic search in any vertical where domain-specific KGs exist but are too costly to query at runtime. The citation-grounded benchmark construction methodology is also directly applicable to building evaluation infrastructure for agentic systems in other scientific domains.