Skip to content

Towards Accurate UAV Image Perception: Guiding Vision-Language Models with Stronger Task Prompts

🕒 Published (v1): 2025-12-08 08:44 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AerialVP is a training-free agent framework that improves VLM performance on UAV imagery by automatically enriching sparse user task prompts with semantic, spatial-position, and spatial-relationship information before passing them to any off-the-shelf VLM. It introduces AerialSense, a 53K-sample UAV perception benchmark spanning visual grounding, reasoning, and VQA tasks.

Problem

VLMs struggle with UAV imagery because brief task prompts fail to achieve semantic alignment with visually dense aerial scenes (dense targets, scale variation, cluttered backgrounds). Training-based alignment approaches are bottlenecked by the scarcity of large-scale UAV image-text datasets and leave the inference-time prompt gap unresolved.

Method

AerialVP is a three-stage prompt enhancement pipeline acting as a plug-in between the user and a downstream VLM. A LLM-based Task Engine orchestrates tool calls over a Tool Repository built on the Model Context Protocol (MCP): 1. Task Prompt Analysis — two parallel tools classify task type (grounding/reasoning/VQA) and extract task-focused object categories; a third tool synthesizes an enhancement plan. 2. Tool Selection — the Task Engine maps each required enhancement subtask to the best-fit model in the repository (e.g., YOLO-World for spatial position, DAM for semantic description, SpaceOm for spatial relationships). 3. Enhanced Prompt Generation — the engine assembles and executes a structured program calling the selected tools sequentially, then fuses outputs via a template that prepends the original instruction and appends multi-dimensional auxiliary descriptions.

The enhanced prompt TÌ‚ = Agent(T, I) is fed with image I to any VLM: R = f_VLM(TÌ‚, I). No VLM weights are modified.

Key Contributions

  • First framing of UAV VLM failure as a prompt-side alignment problem rather than a model-side training problem.
  • AerialVP: first agent framework for task prompt enhancement in UAV perception; training-free, MCP-standardized, extensible tool repository.
  • AerialSense benchmark: 7,119 UAV images, 53,374 test samples across Aerial Visual Grounding, Visual Reasoning, and VQA; sources include AerialVG, refGeo, CODrone, and VisDrone.

Results

The paper reports "stable and substantial performance improvements in both open-source and proprietary VLMs" on AerialSense, but the provided text excerpt does not include specific numerical scores or per-baseline comparisons — those tables appear beyond the truncation point.

Limitations

  • Quantitative results are not present in the supplied text, making independent assessment of magnitude of gains impossible.
  • Pipeline adds latency proportional to the number of tool calls (three external model invocations per query plus LLM orchestration).
  • Tool repository currently covers three enhancement dimensions; edge cases requiring domain tools not yet in the repository fall back to the basic prompt.
  • MCP-based tool invocation introduces an additional infrastructure dependency absent in direct VLM inference.
  • AerialSense is derived from existing datasets repurposed with new annotations; distribution may not fully capture all real-world UAV conditions.

Relevance to Harnesses / Meta-Harnesses

AerialVP is a domain-specific meta-harness: an LLM-orchestrated layer that decomposes a user request, selects tools from a registry, composes a multi-step execution program, and routes enriched inputs to a downstream model — without modifying any underlying model weights. The MCP-standardized Tool Repository pattern mirrors the tool-registry abstraction central to general-purpose agentic harnesses, and the three-stage analyze→select→generate loop is a concrete instantiation of the plan-execute-synthesize cycle. For harness researchers, this paper demonstrates how a domain-specific orchestrator can be layered atop commodity VLMs to achieve task-specific gains, offering a blueprint for modular, swappable-tool harness design in perception-heavy domains.