Skip to content

Rethinking Foundation Model Collaboration: Enhancing Specialized Models through Proxy Task Reasoning

🕒 Published (v1): 2026-06-30 05:35 UTC · Source: Arxiv · link

Why this paper was selected

FM + specialized model collaboration via proxy tasks; key agent architecture pattern

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FAT (Foundation-Model-Augmented Task-Specific Reasoning) reframes VLM integration as proxy-task reasoning rather than direct structured prediction: a specialist generates geometrically valid hypotheses, and a VLM selects or verifies the best one. Instantiated as ProxySelect (Qwen2.5-VL-7B + LoRA), it consistently improves specialized baselines across four tasks at 1.4–1.8× specialist cost, versus 4.3–7.1× for direct VLM regression that still underperforms the specialist.

Problem

Existing strategies integrate foundation models as direct predictors or end-to-end replacements for specialized modules, forcing them to perform precise metric regression, structured decoding, and geometric estimation—tasks where domain-specific models trained on task supervision remain stronger. This misaligns foundation-model capabilities (semantic reasoning, comparative judgment) with their assigned role, and fails to exploit complementary strengths.

Method

FAT decomposes structured prediction into three role-aligned stages formalized as:

\[p_{\text{FAT}}(y \mid x, p) = \sum_{y'} \int p_\eta(x' \mid x, y')\, p_\psi(y' \mid x) \sum_{a \in \mathcal{A}(y')} \mathbf{1}[y = M(y', a)]\, p_\phi(a \mid x, x', y', p)\, dx'\]
  1. Specialist prediction \(G_\psi\): a task-specific model generates a finite candidate set \(\mathcal{C}(x) = \{y^1, \ldots, y^K\}\) in its native output space.
  2. Information-space reconstruction \(R_\eta\): candidates are converted into a foundation-model-readable multimodal representation (image overlays for 2D/3D boxes, map-conditioned trajectory renders, numbered mask regions).
  3. Proxy reasoning \(H_\phi\): the VLM solves a bounded proxy task (selection, verification) over reconstructed candidates, returning an index \(\hat{i} = \arg\max_i P_\phi(i \mid x, x', \mathcal{C}(x), p)\); the output is \(\hat{y} = M(\mathcal{C}(x), \hat{i})\).

The VLM never synthesizes new coordinates or contours. Training uses cross-entropy over the metric-best candidate label \(i^{\text{gt}} = \arg\min_i d(y^i, y^{\text{gt}})\). Candidates that are neither ground-truth proposals are local perturbations of specialist outputs (perturbing center, size, yaw, class, endpoint, etc.), randomly permuted before each sample to prevent positional bias. The concrete instantiation, ProxySelect, uses Qwen2.5-VL-7B with LoRA on two A100 GPUs; specialists are RT-DETR+YOLO (2D), 3D-MOOD (3D), HPNet (trajectory), and Mask2Former+FastSAM (segmentation).

Key Contributions

  • Proposes FAT, a principled framework that reframes foundation-model collaboration as proxy-task reasoning, with a probabilistic formulation decoupling specialist prediction, information-space reconstruction, and bounded proxy decision.
  • Instantiates FAT as ProxySelect, a unified multimodal selection/verification interface mapping boxes, trajectories, and masks into VLM-readable overlays and structured descriptions.
  • Demonstrates consistent specialist improvement and large efficiency advantages over direct VLM regression across four structurally distinct tasks (2D detection, 3D detection, trajectory forecasting, semantic segmentation).
  • Shows data efficiency of the proxy formulation: performance saturates with ~25% of fine-tuning data, and image-based reconstruction is essential (text-only provides minimal gains).

Results

All comparisons use Qwen2.5-VL-7B + LoRA for both the direct-regression baseline and ProxySelect.

  • 2D detection (COCO, AP@0.50): Specialist 0.644 → ProxySelect 0.682; direct VLM 0.554; cost 1.4× vs 4.9× specialist.
  • 3D detection (KITTI, AP): Specialist 0.222 → ProxySelect 0.295; direct VLM 0.148; cost 1.5× vs 5.1×.
  • Trajectory prediction (Argoverse, ADE↓): Specialist 1.339 → ProxySelect 0.812; direct VLM 5.391; cost 1.5× vs 4.3×.
  • Semantic segmentation (Cityscapes, mIoU): Specialist 0.599 → ProxySelect 0.742; direct VLM 0.544; cost 1.8× vs 7.1×.
  • Oracle best-of-\(K\) upper bounds: 0.711 / 0.322 / 0.594 / 0.803, showing remaining headroom in candidate ranking.
  • \(K=10\) candidates at base perturbation scale \(1.0\times\) is optimal; gains saturate beyond that.
  • LoRA + comparative reasoning outperforms heuristic reranking across all tasks (e.g., trajectory ADE 0.980 heuristic vs 0.812 ProxySelect).

Limitations

  • Performance is bounded by candidate-set coverage: oracle gaps remain large (e.g., trajectory ADE 0.812 vs oracle 0.594), indicating the proxy reasoner cannot always identify the best available candidate.
  • The framework inherits specialist failure modes—if the specialist produces no good hypothesis, the VLM cannot recover.
  • Candidate generation relies on hand-designed perturbation schemes per task; generalization to new tasks requires re-engineering the perturbation space and reconstruction representation.
  • All experiments use a single 7B VLM backbone (Qwen2.5-VL-7B); scaling behavior and transferability to other VLMs are untested.
  • Evaluation is restricted to four tasks within autonomous driving / urban perception; generalization to other embodied domains is asserted but not demonstrated.

Relevance to Vision-Language Models

This paper directly addresses how VLMs should be integrated into structured-output pipelines, arguing that VLMs are better comparative reasoners than metric regressors—a distinction increasingly important as VLMs are deployed beyond captioning into perception tasks. The ProxySelect instantiation provides a concrete recipe for using any VLM as a post-hoc hypothesis selector, requiring only LoRA fine-tuning on candidate-selection labels, which is far cheaper than end-to-end VLM training for structured prediction. For researchers tracking VLMs, the work reframes the typical "VLM as predictor" paradigm toward "VLM as reasoner over specialist outputs," with broad applicability to detection, segmentation, and planning. The information-space reconstruction stage (overlays, projections, numbered regions) also illustrates how structured 3D/geometric information can be made accessible to image-language models without modifying their architecture.