Skip to content

LandslideAgent with Multimodal LandslideBench: A Domain-Rule-Augmented Agent for Autonomous Landslide Identification and Analysis

🕒 Published (v1): 2026-06-17 04:00 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LandslideAgent is a three-layer agentic framework for autonomous landslide hazard interpretation that couples a domain-specific fine-tuned VLM (LandslideVLM) with a dual-rule controller to enforce rigorous geoscientific reasoning order during tool invocation. It is anchored by LandslideBench, a new 2,130-sample multimodal dataset of image–mask–text triplets across seven landslide subtypes. The system shifts landslide analysis from static vision-only detection to full-process autonomous inference with structured report generation.

Problem

General-purpose VLMs fail at landslide analysis due to domain hallucinations (e.g., confusing debris deposits with riverbank erosion) and an inability to integrate high-precision specialized tools (segmenters, terrain queries). Existing remote sensing datasets are confined to binary segmentation labels, lacking fine-grained subtype taxonomy and multi-dimensional semantic annotations. General-purpose agents, while capable of dynamic orchestration, skip critical intermediate evidence steps (mask extraction, terrain comparison) because they lack domain-enforced reasoning hierarchies, producing hallucinated or uninterpretable reasoning chains.

Method

Three-layer hierarchical architecture:

  1. Data Support Layer (LandslideBench construction): Geospatial anchors from the Unified Global Landslide Catalogue (UGLC, Cruden–Varnes taxonomy, reliability index 1–3) drive Google Earth RGB image extraction at zoom levels 15–16 (2.4–4.8 m/pixel). Pixel-level masks are annotated via SAM 2 + Labelme human-in-the-loop protocol. Textual descriptions are generated across nine geoscientific dimensions using dual-model parallel inference (GLM-4.6V + Qwen3-VL-Plus); pairs with MPNet cosine similarity < 0.8 are flagged, and all outputs undergo expert audit. Result: 2,130 strictly aligned (image, mask, text) triplets across 7 subtypes and 16 countries.

  2. Cognitive Decision-Making Layer (LandslideVLM): Qwen3-VL-8B-Instruct fine-tuned via LoRA (\(r=8\), \(\alpha=16\)) applied to Q/K/V/O attention projections: \(W_*' = W_* + \frac{\alpha}{r}B_*A_*\). Trained 5 epochs, batch size 8, learning rate \(2 \times 10^{-4}\) with cosine annealing and 5% warm-up.

  3. Collaborative Execution Layer (LandslideAgent): ReAct-style Thought–Action–Observation loop with four components: (a) LandslideVLM as cognitive engine, (b) a tool library (segmentation, boundary refinement, type discrimination, terrain/geological query, surrounding element retrieval, comprehensive assessment), (c) a structured-dictionary memory module for cross-round evidence propagation, and (d) a dual-rule controller enforcing: (i) structured report metadata constraints (mandatory field coverage and analytical sequence in report generation) and (ii) cross-validation identification constraints (requiring evidence from multiple tools before concluding a landslide identification).

Key Contributions

  • LandslideBench: First fine-grained multimodal landslide dataset with 7-subtype labels, pixel-level masks, and nine-dimension geoscientific text descriptions (2,130 triplets); deliberately preserves natural long-tailed class distribution to reflect real prior probabilities.
  • LandslideVLM: Domain-adapted VLM via LoRA on LandslideBench; provides geoscientifically grounded cognitive backbone with reduced hallucination.
  • Dual-rule controller: Novel agent constraint mechanism that injects landslide-domain protocol as hard runtime rules over tool invocation order and evidence requirements, preventing shortcut reasoning.
  • Full-pipeline automation: End-to-end autonomous flow from raw multi-source spatial data to structured geoscientific analytical report via natural language interaction.
  • Baseline establishment: LandslideBench evaluated against five mainstream models on fine-grained classification and semantic segmentation.

Results

  • LandslideVLM vs. base Qwen3-VL-8B-Instruct:
  • Landslide discrimination accuracy: +10.96%
  • Fine-grained classification accuracy: +32.87%
  • Semantic description quality: +15.91%
  • LandslideBench provides baselines across five mainstream models on fine-grained classification and semantic segmentation tasks (specific model names and numbers not reported in the provided excerpt).
  • LandslideAgent enables autonomous multi-source spatial data inference for full-process landslide identification and analysis (quantitative agent-level ablations not included in the provided excerpt).

Limitations

  • Dataset size (2,130 samples) remains modest relative to large-scale vision benchmarks; generalization across rare subtypes constrained by long-tailed distribution.
  • Imagery restricted to Google Earth true-color RGB at zoom levels 15–16; no multispectral, SAR, or DEM modalities in the image track.
  • Dual-rule controller rules are hand-crafted from geoscientific protocols; transferring the framework to other hazard domains (floods, wildfires) requires manual rule re-engineering.
  • Agent-level quantitative ablation results (e.g., with vs. without dual-rule controller) are not fully presented in the available text.
  • Dependence on manual expert audit in the annotation pipeline limits scalability.

Relevance to Harnesses / Meta-Harnesses

LandslideAgent is a concrete example of a domain-constrained tool-orchestration harness: the dual-rule controller acts as a meta-layer that governs which tools the underlying LLM/VLM may invoke and in what sequence, enforcing invariants that the cognitive engine alone cannot reliably maintain. The structured-dictionary memory module is functionally a lightweight state harness — persisting intermediate tool outputs across ReAct cycles and making them available to downstream steps. The tripartite annotation pipeline (dual-model cross-validation + similarity gating + expert audit) used to build LandslideBench is itself a multi-agent verification harness, directly applicable as a pattern for any domain where LLM outputs require quality control at scale. Together these components illustrate how domain rules can be lifted from ad hoc prompting into hard structural constraints within an agentic harness, a key design question for harness/meta-harness research.