Mordal: Automated Pretrained Model Selection for Vision Language Models¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Mordal: automated pretrained model selection for building VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Mordal is an automated framework for selecting the best pretrained vision encoder and LLM combination for a user-defined VLM task, without manual intervention or exhaustive grid search. It combines representation-similarity-based candidate clustering with early stopping and scaling-law-based performance prediction to cut search cost by 8.9×–11.6× over grid search while matching or exceeding its top-1 accuracy.

Problem¶

No single off-the-shelf VLM consistently outperforms others across downstream tasks, yet selecting the right pretrained vision encoder + LLM pair is done by human intuition or brute-force grid search. Existing transferability metrics (EMMS, LogME, LEEP) are designed for unimodal classification/regression and break down for VLMs because zero-shot evaluation is meaningless without vision-text alignment. Exhaustive search is impractical: training one 7B-LLM-based VLM candidate on the LLaVA-1.5 alignment dataset takes >100 GPU hours; a 7×7 grid costs ~5,439 GPU hours total.

Method¶

Mordal addresses the problem along two axes simultaneously:

1. Candidate clustering to shrink search space. - Computes pairwise representation similarity between vision encoders using Centered Kernel Alignment (CKA): \(\text{CKA}(K,L) = \frac{\text{HSIC}(K,L)}{\sqrt{\text{HSIC}(K,K)\cdot\text{HSIC}(L,L)}}\) where \(\text{HSIC}(K,L)=\text{Tr}(KHLH)\). - CKA is task-conditioned (computed on target-domain images) and handles differently-shaped activation matrices from heterogeneous encoders. - Two-step clustering: first cluster vision encoders, then cluster LLMs conditioned on the medoid encoder's image embeddings passed through a warmed-up projector. Candidate clusters are the Cartesian product of the two cluster sets. - Inter-cluster evaluation: train and evaluate only the medoid VLM from each cluster; prune bottom clusters, keep Top-K. - Intra-cluster evaluation: among candidates in Top-K clusters, identify the winner.

2. Efficient evaluation to reduce per-candidate cost. - Early stopping: applies the Successive Halving Algorithm (SHA) during inter-cluster evaluation — all cluster representatives start with budget \(b\), the bottom fraction is eliminated each round, and survivors receive \(b \times \eta\) budget, recycling intermediate checkpoints. - Scaling prediction: exploits an observational scaling law — under fixed model size \(N\), task error decreases log-linearly with log-alignment-data-size \(D\), analogous to \(L(N,D)=\frac{a}{N^\alpha}+\frac{b}{D^\beta}+e\). For each intra-cluster candidate, Mordal fits a linear regression \(f_c\) on a few (Log(\(r\)), Log(Err)) points at decreasing sample ratios \(r\) and predicts full-data performance as \(f_c(1)\), avoiding full training runs.

Early stopping and scaling prediction share intermediate checkpoints to avoid redundant compute.

Key Contributions¶

Formal definition of the pretrained model selection problem for VLMs, distinct from unimodal transferability estimation.
Empirical demonstration that no existing off-the-shelf VLM reliably contains the best components for arbitrary downstream tasks.
Mordal framework: CKA-based two-step clustering + SHA early stopping + observational scaling prediction, all co-designed to reuse checkpoints.
Validation of an observational scaling law between alignment data size and downstream task error for VLMs.

Results¶

Speed vs. grid search: Mordal finds the best VLM 8.9×–11.6× faster across 7 benchmarks; total GPU hours reduced from ~5,439 to 469–607 GPU hours per task.
Ranking quality vs. SOTA: Mordal achieves ~69% higher weighted Kendall's τ than the state-of-the-art model selection method (EMMS) on average across diverse tasks.
Top-1 accuracy parity: Mordal matches grid search's top-1 accuracy on 6 of 7 benchmarks and is within 1.6% on ChartQA (18.6% vs. 20.2%), while the default LLaVA-1.5 (CLIP-Vicuna) consistently underperforms the Mordal-selected model.
Specific top-1 scores found by Mordal: GQA 66.4% (SigLIP-Vicuna), VizWiz 46.9% (SigLIP-Mistral), ScienceQA 78.5% (SigLIP-Llama), AI2D 65.2% (SigLIP-Qwen), MMMU 36.6% (SigLIP-Llama) — all matching grid search optima.
Evaluated on 16 NVIDIA A40 GPUs (48 GB each); 7 vision encoders × 7 LLMs = 49-candidate search space.

Limitations¶

Search space is bounded by a manually curated pool of 7 encoders and 7 LLMs; Mordal does not scale to or auto-filter from the full HuggingFace catalog (150,000+ LLMs).
The observational scaling law is empirically validated but not theoretically grounded; its generalizability to very different alignment data distributions or model architectures is unproven.
Intra-cluster evaluation still requires full training for remaining candidates after SHA pruning, so absolute GPU cost (469–607 hours per task) remains substantial.
CKA computation requires running forward passes on the target-task dataset for all candidate vision encoders, adding non-trivial overhead that is not fully quantified against total savings.
Top-K and threshold hyperparameters (\(t_{ve}\), \(t_{llm}\), \(K\), \(\eta\), \(R\), \(p\), \(\delta\)) require tuning and their sensitivity is only partially ablated.
Only tested on the LLaVA-style architecture (encoder → projector → decoder LLM); applicability to cross-attention or mixture-of-expert VLM architectures is not evaluated.

Relevance to Vision-Language Models¶

Mordal directly addresses the under-explored infrastructure problem of VLM construction: given a downstream task, which pretrained backbone pair (vision encoder + LLM) should be chosen? This is highly relevant because most VLM research assumes a fixed architecture and studies alignment or instruction tuning, while Mordal shows that backbone selection alone can swing task performance substantially (e.g., AI2D: 54.8% → 65.2%). The CKA-based similarity metric is a principled, task-conditioned alternative to heuristic human selection and could generalize as VLM zoos expand. The observational scaling law for alignment data is an actionable finding: it enables cheap performance estimation without full training, a technique that could be adopted in any iterative VLM development pipeline.