CangLing-KnowFlow: A Unified Knowledge-and-Flow-fused Agent for Comprehensive Remote Sensing Applications

🕒 Published (v1): 2025-12-17 09:31 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CangLing-KnowFlow is an LLM-based agent framework for automating end-to-end remote sensing workflows, combining a 1,008-template expert procedural knowledge base (PKB), a runtime dynamic workflow adjustment module, and an evolutionary memory that learns from successes and failures. It is evaluated on KnowFlow-Bench (324 workflows across 162 RS tasks) and surpasses Reflexion by at least 4% task success rate across all complex tasks.

Problem

Existing RS automation is either task-specific fixed pipelines (inflexible), general-purpose LLM agents (prone to planning hallucinations in specialized domains), single-turn tool-call agents (no long-horizon planning), or monolithic VLMs (opaque, no tool orchestration). No prior framework unifies knowledge grounding, runtime adaptability, and experiential learning for end-to-end RS workflows.

Method

The framework has three synergistic components under a central Orchestrator Agent:

Procedural Knowledge Base (PKB): Workflows are modeled as DAGs W = (V, E), where vertices are tool invocations and edges encode precondition/effect compatibility (checked via LLM boolean indicator ϕ_LLM). Tools are formalized with PDDL-style schemas (parameters, preconditions, effects). The PKB contains 1,008 expert-validated hierarchical workflow templates across 162 RS task types, curated via LLM-generated task descriptions followed by manual expert annotation.

Dynamic Workflow Adjustment: On step failure, a two-tier hierarchical repair strategy fires: (Tier 1) query evolutionary memory K_evo for a matching pattern-action repair rule; (Tier 2) if none found, invoke the LLM with full execution history to generate a novel repair. Repair is applied via three DAG manipulations: node replacement, node insertion, or parameter modification.

Evolutionary Memory Module: Post-task, successful adjusted workflows are solidified into new PKB templates (positive feedback); terminal failures yield new IF-THEN pattern-action heuristic rules stored in K_evo (negative feedback). The agent's action policy is formally modeled as an MDP: a_t ← A(H_{t-1}, W_current, K_evo).

Key Contributions

• Expert PKB with 1,008 validated workflow templates across 162 RS task types, formalized as PDDL-schema'd tool libraries (GDAL + deep learning tools for segmentation, detection, change detection, inversion)
• Dynamic runtime workflow adjustment via tiered repair (memory-first, LLM-fallback) and three DAG graph manipulation operators
• Evolutionary memory that converts episodic execution traces into durable procedural knowledge via dual success-solidification / failure-attribution pathways
• KnowFlow-Bench: 324 executable ground-truth workflows (162 tasks × 2 expert annotations each) for holistic evaluation of workflow orchestration, not just tool use

Results

• Surpasses Reflexion baseline by ≥4% Task Success Rate (TSR) across all complex tasks on KnowFlow-Bench
• Evaluated across 13 LLM backbones (open-source to commercial)
• Also evaluated on ThinkGeo benchmark (436 tasks, 14 tools, 7 domains) to test cross-benchmark generalization; framework adapted by restricting action space to ThinkGeo's 14 tools
• Metrics include TSR, First-Pass Accuracy (FPA), Number of Tool Calls (NTC), Number of Interactions (NI), and step-wise process evaluation (GTA-style fine-grained metrics)
• Specific per-backbone or per-task numerical breakdowns are not fully reproduced in the provided text excerpt

Limitations

• PKB curation required manual expert annotation of workflows, limiting scalability to new RS domains without domain expert involvement
• 162 task types cover a finite slice of RS; novel task types outside the PKB incur planning hallucination risk similar to baselines
• Evolutionary memory quality depends on accumulated execution traces; cold-start performance degrades before sufficient experience is accumulated
• Cross-benchmark evaluation required manual environment adaptation (tool-space restriction), indicating portability is not automatic
• No quantitative ablation on the memory module's contribution in isolation is included in the provided text

Relevance to Harnesses / Meta-Harnesses

CangLing-KnowFlow is a textbook domain-specific meta-harness: it wraps heterogeneous tools (GDAL, DL models) into a DAG-driven orchestration loop with explicit knowledge retrieval, runtime monitoring, and self-modifying workflow logic. The three-tier structure—static knowledge retrieval → dynamic replanning → post-hoc memory update—directly mirrors the plan-execute-reflect loop common in general meta-harness architectures. The evolutionary memory mechanism (solidifying successful runs back into the PKB) is particularly relevant as a concrete implementation of harness self-improvement, where the harness's own execution traces become the training signal for future planning. The PDDL-formalized tool schema provides a reusable pattern for how harnesses can enforce tool compatibility constraints programmatically rather than relying on LLM generalization.