SharpZO: Hybrid Sharpness-Aware Vision Language Model Prompt Tuning via Forward-Only Passes

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SharpZO is a two-stage, forward-only prompt tuning method for CLIP-based VLMs that uses sharpness-aware CMA-ES to warm-start zeroth-order (ZO) optimization, producing a flatter loss landscape and stronger initialization before switching to sparse ZO-SGD with a novel Z-pruning strategy. It achieves up to 7% average accuracy gain over state-of-the-art BP-free baselines while converging faster and using fewer forward queries than competitors.

Problem

Gradient-based VLM prompt tuning (e.g., CoOp) requires backpropagation, making it infeasible on inference-only edge devices (IoT ASICs, wearables). Existing BP-free alternatives—evolutionary strategies (ES) and zeroth-order (ZO) optimization—suffer from high gradient-estimation variance, premature convergence, and substantially worse accuracy than first-order methods. Prior variance-reduction efforts focused on parameter dimensionality reduction, which offers limited benefit when trainable prompts are already compact.

Method

SharpZO uses a two-stage all-forward-pass optimization pipeline:

Stage 1 — Sharpness-Aware CMA-ES (warm-up): CMA-ES sampling is shifted by a sharpness perturbation ε* (the SAM worst-case perturbation, estimated via coordinate-wise gradient estimation/CGE rather than RGE to avoid smoothed-objective bias). This biases the population toward flat minima, smoothing the loss landscape and yielding a strong initialization θ_T_c. The Taylor expansion of the expected fitness shows this additionally down-weights high-curvature directions via the covariance update. Stage 1 runs until a validation-accuracy early-stopping criterion is met (typically ≤ 100 steps).

Stage 2 — Sparse ZO-SGD: Fine-grained local optimization uses RGE gradient estimates (q=1 query sufficient after the warm-up) with a masked perturbation vector u·Ω. The sparsity mask Ω is computed via Z-pruning: a second-order Taylor-based sensitivity score that combines weight magnitude with Fisher-approximated Hessian information, then applies Z-score normalization to suppress outlier ZO gradient estimates. The mask is updated every K steps.

Low-dimensional prompt parameters are projected from latent space w ∈ ℝ^d via a fixed random projection A into the CLIP text embedding space.

Key Contributions

• First ZO method to incorporate sharpness-aware initialization (via sharpness-aware CMA-ES) to improve downstream ZO fine-tuning of VLMs.
• CGE-based gradient estimation for computing SAM perturbations without backpropagation, avoiding smoothed-objective bias that would degrade sharpness computation.
• Z-pruning: a ZO-specific sparse mask criterion combining second-order (Fisher) sensitivity with Z-score normalization, outperforming magnitude-based pruning in compact prompt-tuning settings.
• Theoretical convergence analysis (under PL + Lipschitz assumptions) showing that sharpness-aware initialization clips the effective smoothness constant L by ρ and linearly reduces the initial error gap ∆₀.

Results

• 11-dataset few-shot classification (16-shot, CLIP ViT-B/16): SharpZO avg 75.64% vs. ZIP 70.98%, BlackVIP 68.20%, BBT 72.42%, ZS-CLIP 65.23%. Gains are largest on Flowers102 (+15%), EuroSAT (+5.2%), UCF101 (+4.6%), DTD (+9.7%).
• ResNet-50 backbone: SharpZO avg 69.76% vs. ZIP 64.08%, BlackVIP 60.84%, BBT 65.40%.
• Distribution shift robustness (ImageNet variants, ViT-B/16): SharpZO avg 58.9% vs. ZIP 56.9%, BlackVIP 54.9%, CoOp (BP) 59.4% — near-matching BP baseline.
• Time-to-test-accuracy: On EuroSAT, SharpZO reaches threshold in 12.7 min vs. ZIP 251.4 min, BlackVIP 201.5 min; per-step time ~0.0069s vs. ZIP ~0.53s.
• vs. CraFT (BP-based): SharpZO outperforms CraFT on all four reported tasks (e.g., ImageNet 71.60 vs. 68.21, EuroSAT 79.42 vs. 72.07) with lower memory (e.g., 2075 vs. 2179 MB on EuroSAT).

Limitations

• Evaluated exclusively on CLIP (ResNet-50, ViT-B/16, ViT-B/32, ResNet-101); generalization to larger or generative VLMs (e.g., LLaVA-style multimodal LLMs) is left as future work.
• Stage 1 CGE gradient estimation scales as O(d) queries per sharpness step, which becomes expensive if prompt dimensionality d is increased; practical impact limited because Tc ≪ T.
• Change-point Tc between stages requires either manual tuning or an early-stopping heuristic; the automatic criterion (10 consecutive steps without ≥0.01 accuracy gain) is heuristic and task-sensitive.
• All experiments use 16-shot settings; behavior at very low (1-shot) or full-data regimes is not reported.
• Comparison with BP-based methods is acknowledged as limited by deployment constraints, not capability; the accuracy gap to strong BP baselines (e.g., CoOp on several datasets) is not fully closed.

Relevance to Vision-Language Models

SharpZO directly targets the practical deployment bottleneck for CLIP-style VLMs: enabling adaptation on inference-only hardware without any gradient infrastructure. By framing landscape sharpness as an initialization problem rather than a training regularizer, it bridges SAM (a generalization technique from supervised learning) and derivative-free optimization, a connection not previously explored in VLM prompt tuning. For researchers tracking VLMs, this work advances the frontier of parameter-efficient, hardware-agnostic adaptation—relevant to on-device multimodal applications and resource-constrained deployment of foundation models. It also sets a cleaner benchmark against BP-free baselines (reproducing and extending ZIP/BlackVIP comparisons) and provides a convergence theory specific to the ZO prompt-tuning regime.