Skip to content

MED-DSLC: Multi-Expert-Domain Classification via Domain Supervision and Logit Calibration

🕒 Published (v1): 2026-07-13 01:18 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; Nuno Vasconcelos (UCSD); fixes CLIP logit non-comparability across domains—principled zero-shot fix

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs like CLIP are routinely fine-tuned with LoRA for specialist domains, producing fragmented ecosystems of incompatible adapters. MED-DSLC merges these adapters into a unified Multi-Expert-Domain (MED) classifier by combining supervised domain routing with per-domain temperature scaling, restoring the global logit comparability that zero-shot inference requires. On a 9-dataset benchmark with 783 classes it achieves +15% mean accuracy over the LoRA-Mean baseline and outperforms all MoE and model-merging competitors.

Problem

CLIP's zero-shot capability rests on globally calibrated logits: any candidate class can be ranked by softmax over a shared embedding. Parameter-efficient fine-tuning (LoRA) breaks this invariant — each adapter rescales logit magnitudes independently by some implicit \(\gamma_d\). When logits from heterogeneous domain experts compete under a joint softmax over the union label space \(\mathcal{Y} = \bigcup_d \mathcal{Y}_d\), the domain with the highest-magnitude logits dominates predictions for out-of-domain inputs. Existing MoE merging methods (MoLE, Phatgoose) use unsupervised gating and do not address cross-domain logit calibration, causing accuracy to degrade as domain count grows.

Method

MED-DSLC is a frozen-LoRA MoE architecture with two targeted additions:

1. Domain-supervised routing. A gating network \(g(x) = \rho(W_g x)\) computes mixing weights \(\alpha_d^{(\ell)}(x)\) per layer, and the MED-LoRA residual is: $\(\Delta h^{(\ell)}(x) = \sum_{d=1}^{D} \alpha_d^{(\ell)}(x)\, \Delta h_d^{(\ell)}(x)\)$ Rather than relying on an unsupervised load-balancing loss, MED-DSLC exploits the fact that the domain label \(d^* = \arg\max_d \mathbf{1}(y \in \mathcal{Y}_d)\) is trivially derivable from the class label at training time. A domain classification loss supervises the gating directly: $\(\mathcal{L} = \mathcal{L}_{\text{cls}} + \lambda\,\mathcal{L}_{\text{domain}} = -\log p(y|x) - \lambda \log \alpha_{d^*}(x)\)$

2. Domain-wise logit calibration. A learnable scalar temperature \(\tau_d > 0\) is introduced per domain. Before joint softmax over \(\mathcal{Y}\), each class logit is rescaled: $\(\tilde{z}_c(x) = z_c(x)/\tau_d, \quad c \in \mathcal{Y}_d\)$ This preserves within-domain ranking while aligning cross-domain magnitudes, adding only \(D\) scalar parameters without touching the VLM or LoRA weights.

Key Contributions

  • Identifies cross-domain logit miscalibration (heterogeneous implicit scale factors \(\gamma_d\)) and unsupervised gating as the two primary failure modes of MED classifiers.
  • Proposes domain-supervised gating via \(\mathcal{L}_{\text{domain}}\), replacing heuristic load-balancing losses with direct domain classification.
  • Proposes domain-wise temperature scaling (\(D\) learned scalars) to restore globally comparable logits without modifying frozen adapters.
  • Demonstrates that the combination achieves state-of-the-art mean accuracy and scales robustly as the number of domains increases.

Results

All numbers from Table 1, cross-domain protocol, 9 benchmarks (Caltech101, EuroSAT, FGVC Aircraft, Stanford Cars, Oxford Pets, Oxford Flowers, DTD, UCF101, Food101), 783 classes, 16-shot LoRA training, rank-2 LoRA:

  • Base split (seen classes), mean accuracy:
  • MED-DSLC: 85.51% (+15.39% over LoRA-Mean)
  • Expert-LoRA (oracle upper bound): 85.48%
  • KnOTS-DARE-TIES: 78.17%
  • MoLE: 71.80%

  • LoRA-Mean: 70.12%

  • All split (seen + unseen classes), mean accuracy:

  • MED-DSLC: 71.80% (+6.83% over LoRA-Mean)
  • KnOTS-DARE-TIES: 68.66%
  • MoLE: 65.85%
  • LoRA-Mean: 64.97%

  • CLIP (zero-shot): 62.52%

  • Largest per-dataset gains on EuroSAT (+15%) and FGVC Aircraft in cross-domain setting vs. MoLE.

Limitations

  • Domain supervision requires domain labels derivable from class labels during MED training; this assumption holds only when class-to-domain membership is known and non-overlapping (overlapping classes are manually excluded in experiments).
  • Only \(D\) temperature scalars are calibrated; more nuanced per-class or per-layer calibration is not explored.
  • Evaluation limited to 9 fine-grained visual recognition datasets; behavior with tens or hundreds of domains is claimed but not empirically shown at large scale.
  • Requires access to all domain LoRA adapters simultaneously; dynamic addition of new domains would require re-estimating \(\tau_d\) values.
  • Paper text is truncated — ablations and sensitivity analyses (e.g., choice of \(\lambda\), gating non-linearity) are not fully visible.

Relevance to Vision-Language Models

This paper directly attacks a practical bottleneck in deploying CLIP-family VLMs at scale: the breakdown of global logit comparability when domain-specific LoRA adapters are merged. The core insight — that softmax-based zero-shot inference is valid only if logits are globally calibrated, a property that independent fine-tuning destroys — generalizes beyond CLIP to any contrastive VLM relying on cosine similarity logits. The lightweight fix (D learned temperatures + supervised gating) is immediately applicable to any MoE-based adapter composition framework, making it directly relevant to ongoing work on adapter merging, model merging, and multi-task VLM specialization.