SAMPLe: SAM-based Optimizer for Prompt Learning in VLMs

🕒 Published (v1): 2026-07-07 01:21 UTC · Source: Arxiv · Venue: ECCV · link

Why this paper was selected

ECCV; SAM-based optimizer for VLM prompt tuning; directly addresses performance-generalization tradeoff

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SAMPLe is a plug-in sharpness-aware minimization optimizer for VLM prompt learning that addresses the performance-generalization dilemma by enforcing a dual-objective: minimizing training loss while steering the perturbed gradient orthogonal to the full-batch gradient, promoting flat minima without sacrificing accuracy on seen distributions. It integrates into existing prompt learning frameworks (CoOp, CoCoOp, MaPLe, TCP, Co-Prompt) as a drop-in optimizer replacement. Experiments across 11 classification datasets show consistent improvements over SAM, F-SAM, SAGM, and task-specific prompt learning baselines.

Problem

Prompt learning in VLMs (e.g., CLIP) suffers from a performance-generalization dilemma: the small, restricted parameter space of learnable prompts makes optimization highly sensitive, causing convergence to sharp, high-curvature minima that overfit seen classes and fail on unseen ones. Prior SAM-based approaches (F-SAM, SAGM, GCSCoOP) either fail to balance loss value with landscape flatness or rely on manually tuned hyperparameters with limited scalability.

Method

SAMPLe augments the standard SAM objective with a non-SAM perturbation \(\epsilon^P = \alpha(\nabla_{\theta_t} L - \nabla^{\mathcal{F}} L)\) to define the dual objective:

\[\min_\theta \left[ L(\theta; \mathcal{D}) + L\!\left(\theta + \epsilon^\star - \alpha \nabla^{\mathcal{B}} L(\theta; \mathcal{D}); \mathcal{D}\right) \right]\]

where \(\epsilon^\star = \rho \cdot \frac{\nabla L}{\|\nabla L\|_2}\) is the standard SAM perturbation and \(\nabla^{\mathcal{B}} L = \nabla L - \xi\sigma \nabla^{\mathcal{F}} L\) is the batch-specific gradient orthogonal to the full-batch gradient \(\nabla^{\mathcal{F}} L\) (approximated via EMA). Taylor expansion of the second term reveals two forces acting on the perturbed-point gradient \(\nabla L_p\): (1) exploitation — alignment of \(\nabla L_p\) with the mini-batch gradient \(\nabla L\) via \(-\alpha(\nabla L_p \cdot \nabla L)\); (2) exploration — anti-alignment with \(\nabla^{\mathcal{F}} L\) via \(+\alpha\xi\sigma(\nabla L_p \cdot \nabla^{\mathcal{F}} L)\). These forces cancel at equilibrium to drive \(\nabla L_p \perp \nabla^{\mathcal{F}} L\), preserving stochastic variability and targeting flat minima. Learning rate and perturbation radii decay as \(O(1/\sqrt{t})\); convergence is proved at rate \(O(\log T / \sqrt{T})\), matching SGD/Adam. \(\xi\) (cosine similarity) and \(\sigma\) (normalization ratio) are computed analytically each step — no hand-tuned hyperparameters beyond \(\rho\) and \(\alpha\).

Key Contributions

• Introduces SAMPLe, a model-agnostic SAM-based optimizer tailored to the narrow parameter space of VLM prompt learning, requiring no architectural changes.
• Formalizes a dual-objective optimization framework that simultaneously enforces low ERM loss (exploitation) and gradient orthogonality to the full-batch component (exploration), with both constraints satisfied adaptively each iteration.
• Provides convergence proof under bounded, \(K\)-Lipschitz gradient conditions, showing \(O(\log T / \sqrt{T})\) convergence rate.
• Demonstrates integration and gains across five diverse prompt learning frameworks: CoOp, CoCoOp, MaPLe, TCP, Co-Prompt.
• Analyzes failure modes of existing SAM variants (full-batch bias, hyperparameter brittleness) in the prompt learning context.

Results

Evaluated on base-to-new generalization, cross-dataset generalization, and cross-domain generalization across 11 datasets (ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101) plus four ImageNet-variant target domains (ImageNetV2, ImageNet-Sketch, ImageNet-A, ImageNet-R): - On CoOp (ImageNet), SAMPLe achieves ~73.4% vs. baseline ~72.5% and SAGM ~73.0% (from Fig. 3 bar charts; exact per-dataset tables appear to be in the truncated section). - On CoCoOp (ImageNet), SAMPLe reaches ~74.2% vs. baseline ~73.5% and F-SAM ~73.7%. - On MaPLe (ImageNet), SAMPLe reaches ~74.6% vs. baseline ~74.0% and SAGM ~74.3%. - SAMPLe exhibits lower coefficient of variation (CV) across perturbation radius \(\rho\) values than F-SAM and SAGM, indicating greater hyperparameter robustness. - Consistently outperforms SAM, F-SAM, SAGM, and GCSCoOP as plug-in replacements across all five frameworks.

(Full per-dataset numerical tables are in the truncated portion of the paper.)

Limitations

• Requires computing and maintaining an EMA of full-batch gradients (Eq. 9), adding memory and compute overhead per step relative to vanilla SGD/Adam.
• Two additional hyperparameters (\(\alpha\), \(\rho\)) must still be tuned, though \(\xi\) and \(\sigma\) are computed automatically.
• Evaluation is restricted to few-shot classification settings with CLIP; applicability to other VLM architectures or tasks (VQA, dense prediction) is not demonstrated.
• GCSCoOP comparison is limited; the paper notes its scalability issues but the head-to-head performance gap is not fully quantified in the visible text.
• Convergence analysis assumes bounded, Lipschitz-smooth gradients — standard assumptions that may not tightly characterize the prompt learning loss landscape in practice.

Relevance to Vision-Language Models

SAMPLe directly targets the most widely used paradigm for adapting frozen VLMs — soft prompt tuning — and addresses its core failure mode: the restricted parameter subspace amplifies sensitivity to sharp minima, making standard optimizers insufficient. By treating the optimizer itself as the lever for generalization (rather than adding prompt regularization or auxiliary losses), SAMPLe is orthogonal to and composable with architectural improvements like multi-modal prompting (MaPLe) or class-conditioned prompting (CoCoOp). This is directly relevant to the broader VLM prompt learning literature, including zero-shot and few-shot transfer, cross-dataset generalization, and domain shift — all central evaluation axes in the field. The framework's model-agnostic, plug-in nature means it can augment future prompt learning methods without modification.