Skip to content

GISclaw: A Comprehensive Open-Source LLM Agent System for Realistic Multi-Step Geospatial Analysis

🕒 Published (v1): 2026-03-27 09:46 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GISclaw is an open-source LLM agent system that executes professional-grade, multi-step geospatial analysis entirely through the open-source Python stack (GeoPandas, rasterio, scipy, scikit-learn), eliminating dependence on proprietary GIS platforms. It pairs a persistent Python sandbox with three engineered prompt rules and an Error-Memory self-correction module, and supports six heterogeneous LLM backends under a unified interface. On GeoAnalystBench (50 expert tasks, avg. 5.8 steps), GISclaw reaches up to 100% task success; a controlled 1,800-experiment study shows Single-Agent ReAct reliably outperforms Dual-Agent Plan-Execute-Replan on every cloud backend.

Problem

Existing LLM-driven GIS agents suffer from three system-level gaps: (1) narrow data-type coverage—none integrates vector overlays, raster algebra, geostatistical interpolation, and ML classification in a single pipeline; (2) platform and model lock-in, typically coupling to ArcGIS/QGIS and a single LLM (GPT-4), preventing air-gapped deployment and cross-model comparison; (3) fragmented evaluation—binary success, CodeBLEU, or single-dimension LLM judging each miss complementary quality dimensions, and the functional-equivalence problem (multiple valid implementations for one GIS operation) makes lexical metrics unreliable. No prior system isolates the contribution of multi-agent orchestration with a controlled single-agent baseline.

Method

Architecture. GISclaw wraps a pluggable LLMEngine (cloud API or locally served open-weight model) around a persistent Python sandbox pre-loaded with the full open-source GIS stack. The sandbox maintains interpreter state across rounds within a task (Jupyter-like), uses asymmetric output truncation (stdout front-truncated to keep latest results; stderr tail-truncated to preserve root-cause tracebacks), and enforces a 10-minute per-task timeout.

Prompt engineering (three rules). (i) Schema Analysis: the agent's first action is mandatory data inspection (column names, CRS, dtypes, head rows) before generating any analysis code. (ii) Package Constraint: the prompt explicitly lists allowed open-source libraries and redirects from arcpy/pykrige to scipy/rasterio equivalents. (iii) Optional Domain Knowledge Injection: a free-text field lets a domain expert inject procedural hints (calibrated thresholds, step orderings) into the prompt; the system is designed to perform comparably with or without this hint.

Two pluggable agent architectures. Single Agent (SA) follows the ReAct loop (Thought → Action → Observation) augmented with an Error-Memory module that records failed code patterns across rounds to prevent repetition. Dual Agent (DA) uses a Plan-Execute-Replan pipeline: a Planner decomposes the task into 3–7 ordered steps; a Worker executes each in the shared sandbox with up to 10 self-correction rounds per step; on failure the Planner is re-invoked as a Replanner with failure context (up to 2 replanning cycles). Both architectures share the same LLM and sandbox namespace.

Evaluation (three layers). Layer L1 (code structure): CodeBLEU + API Operation F1 (GIS-specific operation set-precision/recall/F1, robust to functional equivalence). Layer L2 (reasoning process): cosine similarity of execution-log embeddings (OpenAI text-embedding-3-large, 3072-d) + LLM-as-Judge on 5 dimensions. Layer L3 (output): vision scoring for PNGs, programmatic raster/vector/tabular comparison. Composite score: $\(S_{\text{comp}} = 0.4 \cdot R_{\text{succ}} + 0.3 \cdot Q_{\text{out}} + 0.15 \cdot F_{\text{api}} + 0.15 \cdot C_{\text{emb}}\)$ where \(Q_{\text{out}}\) averages only over successful tasks to avoid double-penalizing failures already captured by \(R_{\text{succ}}\).

Key Contributions

  • First LLM-driven GIS agent covering the full professional workflow spectrum (spatial overlay, raster algebra, kriging, ML classification, network analysis, cartography) on a purely open-source Python stack with no proprietary dependencies.
  • Backend-agnostic LLMEngine interface supporting six heterogeneous backends (GPT-5.4, GPT-4.1, DeepSeek-V3.2, Gemini-3-Flash, Llama-3.3-70B, Qwen2.5-Coder-14B), enabling air-gapped deployment.
  • Three-layer evaluation protocol unifying code-structure, reasoning-process, and type-specific output verification into a composite score with sensitivity validation (Kendall's \(\tau\) median = 0.94).
  • Open-source rewrite of all 50 GeoAnalystBench gold-standard solutions from ArcPy to open-source Python (released under MIT).
  • Controlled 1,800-experiment study (50 tasks Ă— 6 backends Ă— 2 architectures Ă— 3 runs) with bootstrap 95% CIs, paired Wilcoxon signed-rank tests, and Cliff's \(\delta\) effect sizes.

Results

  • GISclaw (GPT-5.4, SA): up to 100% task success, 97% mean success across three repeated runs on GeoAnalystBench.
  • SA reliably outperforms DA on every cloud backend (Cliff's \(\delta\) = 0.15–0.41); DA offers a gain only for the locally deployed Qwen2.5-Coder-14B.
  • Composite-score sensitivity analysis under varied dimension weights: median Kendall's \(\tau\) = 0.94, confirming ranking stability.
  • Per-task success varies non-trivially across runs even for flagship models, motivating repeated-run evaluation as a default.
  • DeepSeek-V3.2 provides competitive performance at ~\(0.28/\)0.42 per million tokens vs. GPT-5.4 at \(2.50/\)15.00.
  • Ablation: Schema Analysis prompt rule alone eliminated column-name guessing errors and significantly improved the 14B local model's output quality.

Limitations

  • GeoAnalystBench contains only 50 tasks; conclusions about architecture choice and model ranking may not generalize beyond this benchmark.
  • Domain Knowledge Injection (the human-designed workflow hint) is uniformly injected during evaluation, which inflates comparability across models but does not test fully unconstrained free-exploration scenarios.
  • Gold-standard rewrites from ArcPy to open-source were done manually; functional equivalence is asserted but not formally proved for all edge cases.
  • The 10-minute per-task sandbox timeout and the Replanner's 2-cycle cap are empirically chosen hyperparameters whose sensitivity is not fully ablated.
  • Evaluation of LLM-as-Judge uses DeepSeek-V3.2 fixed across all runs; judge model bias is not characterized.
  • Local deployment experiments are limited to two models on specific hardware (multi-GPU server and single RTX 3090); generalization to other hardware configurations is not tested.

Relevance to Harnesses / Meta-Harnesses

GISclaw is a domain-specialized harness whose architecture instantiates the canonical harness design pattern: a model-agnostic LLM interface, a persistent stateful execution environment (the Python sandbox), structured prompt engineering rules as compile-time harness configuration, and an Error-Memory module as a runtime self-repair loop. The backend-agnostic LLMEngine with provider-specific adapters is a textbook meta-harness abstraction—the same agent logic runs against six different model backends without modification, isolating architectural effects from model effects. The paper's central empirical finding—that Single-Agent ReAct outperforms Dual-Agent Plan-Execute-Replan for capable cloud models, with multi-agent complexity only beneficial when model capability is the bottleneck—provides actionable harness design guidance: orchestration complexity should be a function of underlying model strength, not a default. The three-layer composite evaluation protocol, including the API Operation F1 metric designed to handle functional equivalence, is directly transferable to harness evaluation methodology for other code-generation domains.