LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLM-wrapper adapts frozen, black-box Vision-Language Models (VLMs) for Referring Expression Comprehension (REC) by serializing the VLM's bounding-box outputs into natural language and feeding them to a lightly LoRA-fine-tuned LLM that selects the best matching box. The method requires only forward-pass access to the VLM, works across multiple VLM/LLM combinations, and delivers +8.7 to +18.3 P@1 gains over zero-shot VLMs on semantically challenging benchmarks.

Problem

Zero-shot VLMs substantially underperform task-specifically fine-tuned models on REC, yet white-box fine-tuning requires gradient access — infeasible for closed-source or API-only models. Existing black-box adaptation methods either need internal representations (logits, activations) or require multiple inference calls per sample, limiting their applicability to truly opaque systems.

Method

Given a (image, referring expression) pair, the frozen black-box VLM runs a single inference producing labeled bounding boxes with coordinates and confidence scores. LLM-wrapper serializes these detections into a structured natural-language prompt listing each box's index, label, xyxy coordinates, and score, then appends the referring expression and asks the LLM to output the index of the best-matching box. The LLM is fine-tuned with LoRA (rank r=128 by default) using cross-entropy on the box-index targets, trained on (VLM output, ground-truth box) pairs filtered to IoU > 0.5. Box order is randomly permuted during training to prevent shortcut learning. Training uses 4-bit quantization and flash attention, completing on a single 40 GB A100 in under 7 hours. At inference, the method requires only one VLM call and one LLM call per sample.

Key Contributions

• Black-box VLM adaptation pipeline requiring only forward calls — no access to weights, gradients, or logits.
• Prompt construction scheme that converts detection outputs (label, coordinates, score) into natural language for LLM reasoning.
• Demonstration that LoRA fine-tuning of the LLM on VLM outputs suffices; zero-shot LLM gives negligible gain, confirming the fine-tuning step is necessary.
• VLM ensembling: LLM-wrapper unifies variable-length prediction sets from multiple VLMs in a single prompt, yielding +2.1–2.5 P@1 over the best single-model variant.
• Cross-VLM and cross-dataset transfer: an LLM-wrapper trained on one VLM's outputs transfers to a different VLM (e.g., +29.0 P@1 on GD-1.5 subset) and generalizes to unseen datasets without retraining.

Results

• RefCOCOg (test-umd): GD zero-shot 59.32 → LLM-wrapper with Llama 3 8B 77.36 (+18.0 P@1); GDrec zero-shot 68.37 → 78.01 (+9.6 P@1); Flo2 zero-shot 66.16 → 78.03 (+11.9 P@1).
• RefCOCO (test-unc): GD +23.0 P@1 with Llama 3 8B over zero-shot; GDrec +21.0 P@1; Flo2 +14.7 P@1.
• Talk2Car (test): gains range from +8.7 to +18.3 P@1 across all VLM/LLM combinations.
• RefCOCO+ (test-unc): smaller but significant gains (+4.0–12.0 P@1); lower because RefCOCO+ excludes location words, reducing LLM's spatial reasoning advantage.
• HC-RefLoCo zero-shot transfer (Flo2 trained on RefCOCOg): +19.1 P@1 (test) vs. zero-shot Flo2; outperforms white-box Flo2 fine-tuned on RefCOCO/+/g (+8.2 P@1), despite those being exposed to more data.
• VLM ensembling (Flo2+GDrec, Llama 3 8B, RefCOCOg): 81.25 val / 80.13 test, +2.5/+2.1 over best single-VLM adaptation.
• LLM scale ablation: Pearson r=0.88 between HellaSwag score and REC P@1; GPT-Neo 139M yields only +2.2 P@1 vs. +18.0 for Llama 3 8B.
• Performance nearly saturates at 30k training samples (~2h training).

Limitations

• Additional inference latency from the LLM reasoning step on top of the VLM.
• Performance is bounded by VLM recall: if the target box is not among the VLM's proposals (IoU < 0.5), LLM-wrapper cannot recover it.
• Bounding-box coordinates and labels alone may be insufficient when disambiguation requires fine-grained visual features not captured in text (e.g., subtle appearance differences between distractors).
• Effectiveness scales with LLM size; sub-billion parameter LLMs provide only marginal improvement.
• LLM-wrapper is not designed to match white-box fine-tuning; a performance gap remains in most settings.

Relevance to Vision-Language Models

LLM-wrapper addresses a practical but underexplored regime in VLM adaptation: improving closed-source or API-gated models without any gradient access, which is increasingly relevant as capable proprietary VLMs proliferate. The work demonstrates that the reasoning bottleneck for tasks like REC can be decoupled from the perception backbone — a principle applicable beyond REC to any task where a VLM's vision-side is strong but its language-side disambiguation is weak. The cross-dataset and cross-VLM transfer results suggest that what the LLM learns is abstract spatial/semantic reasoning over symbolic scene representations, not VLM-specific artifacts, offering a modular upgrade path for VLM pipelines. This is directly relevant to researchers studying compositional understanding, grounding, and modular VLM architectures.