Skip to content

MultiUAV-Plat: An LLM-Oriented Platform, Benchmark and Framework for Multi-UAV Collaborative Task Planning

🕒 Published (v1): 2026-06-30 03:02 UTC · Source: Arxiv · link

Why this paper was selected

Multi-UAV LLM platform/benchmark — directly useful for comparing multi-agent harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MultiUAV-Plat is a three-component system—simulation platform, benchmark, and agent framework—for systematically evaluating LLM agents on multi-UAV collaborative task planning. The benchmark enforces partial observability through restricted RESTful APIs and scores agents via hidden server-side validators rather than conversational or QA evaluation. The proposed Agent4Drone framework achieves a 57.9% task pass rate, nearly doubling ReAct's 30.6%.

Problem

Existing UAV simulators expose physics and low-level control interfaces unsuitable for LLM agents, while existing LLM-agent benchmarks lack aerial-robotics constraints such as partial observability, UAV role assignment, closed-loop API interaction, and hidden mission-level validation. No executable benchmark existed for reproducibly comparing LLM agents on multi-UAV collaborative task planning under realistic information restrictions.

Method

Platform (MultiUAV-Plat): A three-layer (Model → Control → Service) simulation environment exposing concise RESTful APIs. A role-based security mechanism enforces agent-level information boundaries: agents receive only percepts within each UAV's local perception radius, never global state. Optional pygame 2D and 3D visualization layers support debugging. Sessions, tasks, and entities are templated for rapid scenario construction.

Benchmark: 75 mission sessions × 20 tasks/session = 1500 natural-language tasks across three scenario families: Area Search, Area Assignment and Patrol, and Target Assignment (25 sessions × 500 tasks each). Difficulty spans five levels (Easy→Extreme) with five independently seeded sessions per level. Tasks carry 9396 hidden server-side validation checks (median 4, mean 6.26 per task, range 1–39), covering 21 endpoint types (movement, perception, target, area, state, coordination). Grading is decoupled from task execution: agents never see the validator logic.

Agent4Drone Framework: A task-specific LLM agent pipeline with six structured modules—memory, observation, task understanding, planning, execution, and verification—applied in a closed loop. The framework structures tool-call management and multi-UAV coordination prompts to prevent the model from conflating high-level planning with low-level API execution.

Key Contributions

  • MultiUAV-Plat: open-source, lightweight LLM-agent-oriented multi-UAV simulation platform with RESTful APIs, role-based partial observability, and optional 2D/3D visualization.
  • MultiUAV-Plat Benchmark: first executable benchmark for LLM-driven multi-UAV collaborative task planning; 75 sessions, 1500 NL tasks, 9396 hidden validation checks across three scenario families and five difficulty levels.
  • Agent4Drone: reference LLM agent framework with six structured modules demonstrating the harness usage, with full paired benchmark comparison against ReAct.
  • Separation of task-solving from task-grading via hidden platform-side validators, enabling reproducible, agent-agnostic scoring.

Results

  • Agent4Drone task pass rate: 57.9% vs. ReAct 30.6% (+27.3 pp).
  • Agent4Drone average task check pass rate: 74.6% vs. ReAct 47.9% (+26.7 pp).
  • Agent4Drone global check pass rate: 72.0% vs. ReAct 43.1% (+28.9 pp).
  • Failed task rate: Agent4Drone 12.9% vs. ReAct 32.4% (−19.5 pp).
  • Baselines compared: ReAct (only baseline reported in the available text).

Limitations

  • Only one baseline (ReAct) is included in the full paired comparison; stronger baselines (e.g., code-generation agents, tree-of-thought variants) are not evaluated.
  • Simulation-only; no sim-to-real transfer experiments are reported.
  • Physics fidelity is deliberately low (lightweight simulator), so dynamics-sensitive tasks cannot be evaluated.
  • Task text is truncated before scenario-family difficulty details are fully described; exact per-family or per-difficulty breakdown results are not available in the provided excerpt.
  • Agent4Drone's performance depends on the underlying LLM backbone, which is not systematically varied in the reported results.

Relevance to Harnesses / Meta-Harnesses

MultiUAV-Plat is a canonical domain-specific evaluation harness: it wraps a simulator with a standardized API layer, injects hidden validators that grade agent behavior independently of the agent's self-report, and enforces information boundaries to prevent agents from short-circuiting the task. The platform–benchmark–framework triad maps directly onto the harness pattern of substrate + evaluation protocol + reference agent, making it directly comparable to harnesses like AgentBench, ALFWorld, and WebShop but targeting aerial robotics. The hidden server-side validation mechanism—decoupling task grading from task execution—is a key harness design principle that prevents agents from gaming evaluation through incorrect but plausible-sounding outputs. Researchers building meta-harnesses that aggregate across domains should note MultiUAV-Plat's RESTful API abstraction as a reusable model for wrapping new simulators without exposing low-level internals to LLM agents.