Skip to content

AgentFAIR: A Multi-Agent Collaborative Framework for FAIRness Evaluation of Geospatial Datasets

🕒 Published (v1): 2026-07-17 09:32 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentFAIR is a multi-agent LangGraph harness that evaluates geospatial datasets against all 13 FAIR sub-principles by orchestrating 13 specialized LLM evaluators, a hybrid metadata extractor, and a critic agent that applies evidence-sufficiency and cross-principle consistency checks with targeted retry. Evaluated on 50 datasets across 10 repositories, it achieves 89% sub-principle agreement across repeated runs and Fleiss' \(\kappa = 0.71\) against expert consensus at ~$0.054/dataset.

Problem

Existing automated FAIR evaluators (F-UJI, FAIR-Checker, FAIRshake, FAIR-enough) use incompatible rubrics, fail on JavaScript-rendered pages, and lack semantic judgment over domain-specific geospatial standards (ISO 19115, EPSG CRS codes, OGC endpoints). Cross-tool score variance on 50 geospatial datasets averages 15.0 percentage-point standard deviation and reaches 30.3 points on a single dataset, making stewardship decisions tool-dependent and unreliable.

Method

AgentFAIR implements a three-stage evidence-first pipeline encoded as a LangGraph directed state machine:

  1. Stage 1 – Extraction: A Playwright crawler renders JavaScript-heavy landing pages; a hybrid extractor combines rule-based parsing of JSON-LD/Schema.org/DCAT fragments with LLM-assisted normalization to produce a provenance-linked metadata record \(M\).

  2. Stage 2 – Parallel evaluation: Four concurrent LangGraph subgraphs (one per FAIR dimension F/A/I/R) each run their sub-principle agents in parallel. Each of the 13 agents produces a tuple \((s_p, e_p, c_p)\) where \(s_p \in \{0,1,2,3\}\) is a maturity score on a rubric-defined ordinal scale, \(e_p\) is a provenance-linked evidence bundle, and \(c_p \in [0,1]\) is a confidence estimate.

  3. Stage 3 – Critic QA: A critic agent checks evidence sufficiency (non-zero scores must cite traceable evidence) and cross-sub-principle consistency (e.g., high A1.1 + low A1.2 flags ambiguous access conditions). If \(c_p < \theta_p\) or a hard deterministic check fails, a targeted retry is issued with refined instructions while caching deterministic signals. Scores aggregate as \(\text{Score}(d) = \frac{1}{3|P_d|}\sum_{p \in P_d} s_p \times 100\).

Geospatial-specific indicators (EPSG codes, ISO 19115-1 elements, OGC service endpoints, GCMD/CF conventions) are explicitly admitted as evidence for I1–I3 and R1.3.

Key Contributions

  • First LLM-driven multi-agent FAIR evaluator with geospatial domain awareness (ISO 19115, OGC, EPSG).
  • 0–3 maturity rubric operationalized per all 13 FAIR sub-principles with explicit geospatial indicators.
  • Critic-based feedback loop with hard checks and confidence thresholds; ablation shows critic improves repeated-run agreement from 71% to 89%.
  • Cross-repository empirical study: 50 geospatial datasets, 10 repositories, comparison against 4 baselines.
  • Fully auditable output pipeline: Markdown report, JSON assessment, SQLite evidence store, local execution traces, optional LangSmith export.

Results

  • Mean FAIR scores (50-dataset sample): Findability 79.7%, Accessibility 70.4%, Interoperability 45.3%, Reusability 72.0%.
  • Rank correlations with four baselines: \(\rho = 0.31\) (FAIR-Checker) to \(\rho = 0.61\) (F-UJI); FAIRenough correlation not statistically significant.
  • Repeated-run consistency (10-dataset subset): 89% sub-principle agreement (SD = 3 pp); 71% without critic (ablation).
  • Expert alignment (15-dataset study): Fleiss' \(\kappa = 0.71\); 82% sub-principle agreement with expert consensus.
  • API cost: ~USD $0.054 per dataset using GPT-4o-mini (temperature 0.1).

Limitations

  • Benchmark limited to 50 datasets; authors explicitly disclaim long-tail generalization.
  • Incomplete component ablations (e.g., no temperature 0.0 vs 0.1 comparison; no full metadata-extractor or Playwright ablation).
  • Single-model-family validation (GPT-4o-mini only); critic and evaluator share the same model, so shared-model bias is not removed.
  • Expert study is preliminary (15 datasets) and does not constitute ground-truth accuracy validation.
  • Does not inspect downloaded data files (no GDAL-based analysis); CRS recognition limited to metadata exposure.
  • No evaluation of retry budget exhaustion behavior or edge cases from navigation timeouts.

Relevance to Harnesses / Meta-Harnesses

AgentFAIR is a textbook domain harness: a LangGraph state machine that decomposes a complex auditing task into 13 specialized sub-agents, runs them in parallel fan-out within four concurrent subgraphs, and then applies a critic meta-agent to enforce cross-agent consistency before aggregating outputs — the critic-with-retry pattern is precisely the quality-gate loop seen in general meta-harness architectures. The framework's design choices (confidence thresholds as routing conditions, deterministic signal caching to stabilize retries, multi-format artifact materialization, optional tracing via LangSmith) are reusable harness-engineering decisions applicable beyond FAIR evaluation. The explicit separation of extraction, evaluation, and QA stages, each with defined input/output contracts, mirrors the pipeline-phase decomposition common to research meta-harnesses.