Skip to content

DroneFINE: Domain-Aware Parameter-Efficient Fine-Tuning of Vision-Language Detectors for Drone Images

🕒 Published (v1): 2026-07-01 02:27 UTC · Source: Arxiv · link

Why this paper was selected

Domain-aware PEFT for VLM detectors on UAV imagery; parameter-efficient adaptation recipe

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DroneFINE is a parameter-efficient fine-tuning (PEFT) framework for adapting Vision-Language Model (VLM) detectors to UAV/drone imagery, targeting the mismatch between "natural-scene, foreground-dominant" VLM priors and "bird's-eye-view, background-dominant, small-object" aerial data. It introduces two complementary modules—HyperAdapter and SemanticGate—that together match full fine-tuning performance while updating far fewer parameters.

Problem

Standard PEFT methods (LoRA, Adapter, etc.) fail to close the domain gap when adapting VLM detectors to UAV imagery for two structural reasons: (1) the adaptation operates in a low-rank space insufficient for UAV imagery's complexity—SVD analysis of FFN layers shows high rank is required, especially in deeper layers; and (2) background-dominant scenes flood the model with irrelevant spatial responses that low-rank adapters cannot selectively suppress. Full fine-tuning is impractical due to computational cost, overfitting on limited domain data, and catastrophic forgetting.

Method

DroneFINE builds on GroundingDINO and inserts two modules:

HyperAdapter replaces static low-rank adapters with a data-dependent, foreground-aware mechanism. \(M\) learnable foreground queries attend over flattened input features \(\mathbf{x}^{\mathrm{flat}}\) via cross-attention to compute attention weights \(\mathbf{A} \in \mathbb{R}^{B \times M \times HW}\), aggregating a global foreground representation \(\mathbf{z}_i = \frac{1}{M}\sum_{j=1}^{M}\sum_{k=1}^{HW}\mathbf{A}_{i,j,k}\cdot\mathbf{x}^{\mathrm{flat}}_{i,k}\). A shared hypernetwork \(H\) maps \(\mathbf{z}\) to dynamic depthwise-separable convolutional kernels \(\mathbf{W}_{dyn}\) organized as parallel \(1\times1\), \(3\times3\), and \(5\times5\) branches (DynConv), enabling multi-scale object handling. Group-shared parameters and compressed hidden dimensions keep the overhead minimal.

SemanticGate exploits VLMs' open-vocabulary text branch to actively suppress background. A 20-item background vocabulary (e.g., "road", "building") is constructed via a four-stage pipeline: UAV image sampling → LLM (GPT-4) keyword generation → empirical detectability validation with DINO-X → manual review. Pre-computed background text embeddings \(\mathbf{F}_\text{back}\) are used to compute per-patch background similarity scores \(\mathbf{s}_{bg}\), which interact with foreground scores \(\mathbf{s}_o\) through an LSTM-inspired three-gate mechanism—Protection Gate (PG), Inhibition Gate (IG), and Candidate Adjustment (CA)—to produce an adjusted similarity matrix \(\mathbf{S}_{adj} = \mathbf{S}_{orig} \odot \mathbf{g}_{pg} + \mathbf{a}_{adj} \odot \mathbf{g}_{ig}\) that re-ranks GroundingDINO's query selection toward foreground regions. An InfoNCE-based contrastive loss \(\mathcal{L}_{con}\) pushes foreground and background text embeddings apart to ensure gate operability.

Key Contributions

  • DroneFINE: a domain-aware PEFT paradigm for VLM-based UAV detectors addressing the bird's-eye-view / background-dominant domain gap.
  • HyperAdapter: hypernetwork-driven, foreground-query-conditioned dynamic multi-path convolutional adapter that overcomes low-rank representation constraints without static structural limits.
  • SemanticGate: text-conditioned background suppression via a curated background vocabulary and LSTM-style gating over GroundingDINO's language-guided query selection, reinforced by InfoNCE contrastive loss on text embeddings.
  • Empirical demonstration that standard PEFT methods (LoRA, Adapter, Mona, NormTuning, VPT, CoOp, Shine) all fall short on UAV benchmarks, motivating domain-specific PEFT design.

Results

  • VisDrone vs. PEFT baselines: DroneFINE-T (1.54M trainable params) achieves 38.4% mAP / 61.2% mAP50, versus best PEFT competitor Mona at 37.9% / 60.5%; surpasses LoRA by +1.7% mAP and +1.9% mAP50; matches full fine-tuning (38.4% / 61.3%) with only 1.54M vs. 27.5M trainable parameters.
  • UAVDT vs. PEFT baselines: DroneFINE-T achieves 26.5% mAP / 43.6% mAP50, outperforming best PEFT competitor Shine (24.3% mAP50) and LoRA (25.0% mAP / 40.8% mAP50); exceeds full fine-tuning (22.4% mAP / 38.6% mAP50).
  • VisDrone vs. UAV-specialized models: DroneFINE-T (Swin-T) reaches 38.4% mAP / 61.2% mAP50, competitive with DroneFINE-B (38.8% / 61.9%) and DroneFINE-L (42.3% / 65.8%); outperforms traditional UAV detectors (UFPMP-Det: 36.6% / 62.4%) and VLM competitors (LAE-DINO: mAP50 56.4%, RT-OVAD: 64.6% mAP50).
  • UAVDT vs. UAV-specialized models: DroneFINE-T reaches 26.5% mAP / 43.6% mAP50, surpassing LAE-DINO (36.5% mAP50) and YOLO-World (35.8% mAP50) and approaching RT-OVAD (40.5% mAP50), despite not using the large-scale LAE-1M pretraining data that competing VLM methods rely on.
  • Zero-shot baseline (GroundingDINO, no fine-tuning): 16.5% mAP / 26.9% mAP50 on VisDrone, confirming the severe domain gap.

Limitations

  • Background vocabulary (20 items) is manually curated and fixed; it may not generalize to UAV scenes significantly different from the streetscape/terrain/atmospheric contexts sampled during construction.
  • SemanticGate relies on GPT-4 for candidate keyword generation, introducing an external dependency in the setup pipeline.
  • The method is evaluated only on VisDrone and UAVDT; generalization to other aerial domains (e.g., maritime, agricultural) is not demonstrated.
  • DroneFINE-L still lags behind RT-OVAD (64.6% vs. 65.8% mAP50 on VisDrone), suggesting room for improvement at the top end of the performance range.
  • The paper does not report latency or throughput numbers, making inference-time cost comparisons with standard PEFT methods unclear.

Relevance to Vision-Language Models

DroneFINE directly addresses a critical failure mode of VLM detectors—specifically GroundingDINO-family models—when deployed outside their natural-image pretraining distribution, showing that standard PEFT recipes are architecturally insufficient for large domain shifts. The hypernetwork-conditioned dynamic adapter and text-conditioned background suppression paradigm represent generalizable PEFT design principles applicable to any dual-encoder VLM detector facing foreground/background imbalance. The SemanticGate's use of the VLM's own text branch to actively suppress non-target regions is a novel inversion of the typical open-vocabulary detection pipeline, relevant to the broader VLM adaptation literature. For researchers tracking VLMs in specialized domains, this paper provides a well-motivated template for diagnosing PEFT bottlenecks via rank analysis and addressing them with domain-aware, modality-specific modules.