Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs

🕒 Published (v1): 2026-07-17 17:59 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UAV-DualCog is a benchmark for evaluating MLLMs on aerial multiview spatio-temporal reasoning, structured around a "dual-cognition" framework that jointly assesses self-state awareness (the UAV's own pose and motion) and environment-state awareness (external scene understanding). Built via an automated pipeline over semantic point clouds from a UAV simulator, it comprises 4,096 image and 2,048 video QA samples across six task types. Extensive evaluation of 30+ proprietary and open-source MLLMs reveals persistent failure on self-state reasoning, viewpoint transformation, spatial grounding, and temporal interval localization.

Problem

Existing UAV-oriented MLLM benchmarks focus on scene understanding, event recognition, or navigation completion and treat the UAV as a passive observer — they ignore the agent's own state (viewpoint, pose, motion) as an explicit reasoning target. No prior benchmark jointly requires both self-state cognition and environment-state cognition with structured spatial/temporal evidence outputs (bounding boxes, temporal intervals), leaving a critical gap in evaluating embodied UAV agents.

Method

Dual-cognition formulation. Each observation \(o\) is associated with a cognition target \(\mathcal{C}(o) = \{c_s, c_e\}\) where \(c_s\) is self-state and \(c_e\) is environment-state. Six tasks operationalize this split across image and video modalities:

• Image tasks (self-state): Self-relative position reasoning (UAV position relative to landmark) and future observation prediction (predict view after a candidate action).
• Image tasks (environment-state): Landmark-relative direction reasoning and landmark-driven action decision (which direction to fly to approach target).
• Video tasks: Flight behavior recognition + temporal localization (\(c_s\)); landmark visibility counting + interval reasoning (\(c_e\)).

All tasks require structured JSON outputs — a discrete answer plus a normalized bounding box (image) or temporal interval (video) — so models that answer correctly without localizing still score low on grounding metrics.

Construction pipeline. Four automated stages: (1) coverage-aware UAV pose sampling in AerialVLN simulator scenes fused into scene-level semantic point clouds with class/instance labels; (2) landmark aggregation, multi-view association, manual verification, and semantic annotation; (3) hierarchical flight behavior trajectory generation (low-level atomic classes: circular, square, triangular, spiral orbits/inspections, gradual approach/depart, S-shaped scan, etc.; high-level composites) with collision and visibility validation; (4) controlled viewpoint sampling and QA generation.

Evaluation metrics. Image: answer accuracy, bounding-box accuracy at IoU \(\geq 0.5\), and mean IoU. Video: task-level and atomic behavior accuracy, visibility-count accuracy, temporal localization accuracy at tIoU \(\geq 0.5\), and mean tIoU.

Key Contributions

• UAV-DualCog benchmark: 12 scenes, 512 landmarks, 4,096 image + 2,048 video samples covering 6 task types, all requiring structured spatial/temporal grounding outputs alongside discrete answers.
• Scalable automated construction pipeline over scene-level semantic point clouds with verifiable pose, visibility, and grounding annotations.
• Systematic evaluation of 30+ models (proprietary, open-source, spatially-specialized), revealing that self-state cognition is consistently harder than environment-state cognition, and that correct discrete answers do not imply reliable localization.
• UAV-DualCog-Train (disjoint scenes), with a lightweight optimization probe demonstrating the benchmark data provides useful structured supervision for UAV-oriented MLLMs.
• Human baseline and thinking/frontier model validation confirming the benchmark is humanly solvable but challenging for current models.

Results

• Answer accuracy vs. grounding gap: Proprietary models achieve moderate answer accuracy but substantially weaker spatial grounding; e.g., Gemini 3 Flash achieves up to 56.2% self-aware answer accuracy but near-zero IoU@50 (0.7%).
• Best discrete accuracy (image tasks, proprietary): GPT 5.5 leads on several tasks — e.g., 65.4% on self-relative position and 53.4% on future observation.
• Best grounding (image tasks): Mimo v2.5 (open-source) achieves 43.2% IoU@50 on self-relative position; Qwen 3.5-27B reaches 42.8% IoU@50.
• Self-state vs. environment-state: Environment-state cognition is consistently stronger than self-state cognition on image tasks across model families.
• Specialized spatial models underperform: SpaceR, SpaceThinker, SpaceOm, ViLaSR, and VST-7B series show competitive answer accuracy but near-zero grounding (IoU@50 \(\approx 0\)–3%), indicating their spatial fine-tuning does not transfer to the UAV aerial domain.
• Video tasks: Behavior understanding, atomic action recognition, and visibility-interval localization are frequently inconsistent across models.
• Parsing audit: A portion of near-zero grounding scores was partly due to format incompatibilities; corrected diagnostic mIoU for Gemini 3.1 Flash Lite rises from 1.1% to 32.6%, but substantial grounding deficit remains.

Limitations

• Simulation-based construction introduces a sim-to-real gap; reducing this gap is explicitly noted as future work.
• All evaluations use Instant mode (no chain-of-thought/thinking), which may underestimate model capability in non-latency-critical settings.
• The benchmark uses a moderate scale (6,144 total samples) and a single simulator (AerialVLN), limiting scene diversity relative to real-world UAV deployments.
• Video tasks are currently limited to two types; richer temporal reasoning scenarios are not yet covered.
• Open-ended and free-form reasoning is only partially evaluated via auxiliary experiments, not as primary metrics.

Relevance to Vision-Language Models

UAV-DualCog directly stresses a capability that general VLM benchmarks ignore: first-person egocentric reasoning where the observer's pose is a primary inference target, not background context. The strong grounding gap (high discrete accuracy, low IoU) revealed across GPT-5, Gemini, Qwen, and InternVL families shows that current VLMs decouple language-level scene description from precise spatial localization — a known open problem in VLM grounding research. The benchmark's structured output requirement (JSON with bounding boxes and temporal intervals) provides a rigorous testbed for evaluating spatial and temporal grounding jointly, complementing existing VQA and referring expression benchmarks. The finding that spatially fine-tuned models (SpaceR, ViLaSR, VST) do not transfer to aerial egocentric reasoning suggests that domain-specific VLM training remains necessary and that aerial embodied intelligence is a distinct frontier for the VLM community.