Skip to content

VICoT-Agent: A Vision-Interleaved Chain-of-Thought Framework for Interpretable Multimodal Reasoning and Scalable Remote Sensing Analysis

🕒 Published (v1): 2025-11-25 09:00 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VICoT-Agent is a multimodal agent framework that interleaves visual tool invocations directly into a chain-of-thought via a dynamic reasoning stack, replacing the token-expensive plan–replan loop. It achieves \(O(T)\) context complexity versus \(O(T^2)\) for plan–replan architectures, and a Reasoning Stack Distillation method transfers the behavior to a 14B model running at 12 GB (AWQ 4-bit). Evaluated on five remote sensing benchmarks, VICoT consistently outperforms RS-specific MLLMs.

Problem

Existing multimodal agents for remote sensing face two coupled failures: (1) plan-execute architectures re-feed full history at every step, causing quadratic token growth and opaque reasoning chains; (2) they depend on model-intrinsic visual capabilities or large-scale pretraining for tool invocation, making them infeasible on edge/satellite hardware requiring \(\leq\)16 GB VRAM.

Method

Stack-based reasoning. At turn \(t\), the agent maintains a stack \(S_t = [s_1, \ldots, s_t]\) where each frame \(s_t = (\phi_t, m_t, e_t)\) encodes the LLM decision \(\phi_t\), a tool-match pair \(m_t = \langle \tau_i^t, a_i^t \rangle\), and evidence \(e_t\) (tool output converted to text). Only the top frame is fed to extend reasoning; context grows \(O(k)\) with sliding window \(k \ll T\) rather than \(O(T^2)\). When tool choice is ambiguous, the top frame is duplicated to spawn a parallel-stack pool of width \(W_t\); stacks are scored and pruned by a lightweight LLM heuristic.

MCP tool suite. All 10 tools (open-vocabulary detector, crop, super-resolution, binarization, cloud/rain removal, denoising, motion deblurring, web search, RAG) are encapsulated via Model Context Protocol. The LLM emits compact XML blocks (<use_mcp_tool>, <server_name>, <tool_name>, <arguments>), enabling uniform one-to-many retrieval without format-specific pretraining.

Reasoning Stack Distillation. GPT-4o-driven VICoT generates 4,400 multi-turn reasoning trajectories (avg. 5–7 steps; ~2,184K tokens total) from 364 HRSC images, forming VICoT-HRSC. Qwen3-14B is fine-tuned on these trajectories; AWQ 4-bit quantization yields a 12 GB model deployable on 16 GB VRAM with 10–15 s/image latency.

Region-Aware Captioned Prompting (UHR head). Ultra-high-resolution images are tiled; GroundingDINO filters empty tiles; each valid tile runs the full VICoT cycle; region-level summaries tagged with spatial coordinates are aggregated into a UHR Integration Prompt for final holistic synthesis.

Key Contributions

  • First multimodal agent framework with explicit multi-round vision-language interleaved reasoning via a stack structure, reducing context complexity from \(O(T^2)\) to \(O(T)\).
  • MCP-based tool encapsulation enabling format-agnostic, model-size-independent, plug-and-play tool invocation.
  • VICoT-HRSC dataset (4,400 trajectories) and Reasoning Stack Distillation enabling 14B models to perform multi-turn structured reasoning without large-scale pretraining.
  • Region-Aware Captioned Prompting head for UHR remote sensing images with spatial tag preservation.

Results

  • Trajectory quality (VICoT-HRSC): VICoT (GPT-4o) achieves 92.3% accuracy vs. GPT-4o baseline at 68.8% (+23.5 pp); human score 95.7 vs. 65.9 for GPT-4o; BLEU 0.95 vs. 0.73.
  • Distilled model: VICoT (Qwen3-14B distill) reaches 88.6% accuracy (<4 pp drop from teacher) and BLEU 0.98 (+0.03 over teacher).
  • RSVQA-HR average: VICoT (4o) 92.25% vs. GPT-4o 75.34% (+16.91 pp); distill 89.09% (+13.75 pp).
  • RSVQA-LR average: VICoT (4o) 94.00% vs. GPT-4o 89.15% (+4.85 pp).
  • MME-RealWorld-RS: Distill 47.40% vs. GPT-4o baseline 28.92% (+18.48 pp), strongest result among all compared models.
  • Efficiency vs. Plan–Replan: −65% token consumption, −48% latency at similar accuracy.

Limitations

  • VICoT-HRSC distillation dataset is small (364 images, 4,400 trajectories) and domain-specific to remote sensing; generalization to other visual domains is unvalidated.
  • Parallel-stack branching width \(W_t\) and stack pruning heuristic are not ablated; their sensitivity is unknown.
  • The paper does not evaluate tool failure modes or robustness when individual MCP tools (e.g., cloud removal) produce erroneous outputs that propagate through the stack.
  • Comparison baselines (VisionGPT, ViperGPT, HuggingGPT, VisualSketchpad) are older architectures; no comparison against more recent agentic frameworks (e.g., ReAct variants with efficient KV caching).

Relevance to Harnesses / Meta-Harnesses

VICoT is a concrete instantiation of a reasoning harness where the orchestration layer (the stack + MCP dispatcher) is architecturally separated from both the reasoning model and the tool implementations — a canonical harness design. The Reasoning Stack Distillation pipeline is itself a meta-harness: a teacher agent (GPT-4o inside VICoT) generates structured execution traces that are used to train a student harness, making harness behavior transferable without rewriting the orchestration logic. The MCP-based tool encapsulation directly addresses a core harness engineering problem: how to expose heterogeneous external tools through a uniform interface that any model can drive. For researchers building general-purpose agent harnesses, the stack-as-context-manager pattern and the distillation-from-traces approach are both directly applicable beyond the remote sensing domain.