Skip to content

Efficient PEFT Methods with Adaptive Checkpointing for Vision Models and VLMs on Resource Constrained Consumer-GPUs

🕒 Published (v1): 2026-07-02 13:31 UTC · Source: Arxiv · link

Why this paper was selected

PEFT comparison for VLMs on consumer GPUs; practical fine-tuning on constrained hardware

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks five PEFT methods (Full FT, LoRA, AdaLoRA, QLoRA, BitFit) across Transformer and Mamba vision backbones under a strict 2 GB VRAM budget, proposes a memory-budget-aware adaptive gradient checkpointing algorithm, and evaluates whether task-specific fine-tuning is competitive against zero-shot VLM baselines (OpenCLIP, SigLIP, DINOv2, PaliGemma, MobileVLM, SmolVLM) on accuracy–energy trade-offs. The adaptive checkpointing selectively recomputes only layers exceeding a runtime memory threshold \(\tau \cdot M\), avoiding the full overhead of static per-layer checkpointing. A key finding is that DINOv2 outperforms all fine-tuned models on CIFAR-100 at a fraction of the energy, while autoregressive VLMs remain uncompetitive.

Problem

Most PEFT benchmarks rank methods solely on accuracy and trainable-parameter count, ignoring memory, energy, and latency on consumer-grade hardware; they also cover only Transformer backbones, leaving the Transformer-vs.-Mamba efficiency trade-off unexplored. Static gradient checkpointing is applied uniformly (all-or-nothing per layer), incurring unnecessary compute overhead when memory is available. Finally, it is unclear whether task-specific PEFT fine-tuning remains justified given the rapid improvement of training-free VLM baselines under matched energy budgets.

Method

Architectures × PEFT grid. Four backbones (ViT-Small/16, TinyViT-21M, Vim-Small, MambaVision-Tiny; all 21–32 M parameters) are crossed with five PEFT methods — Full FT, LoRA (\(r{=}8, \alpha{=}16\)), AdaLoRA, QLoRA (4-bit NormalFloat + LoRA adapters), BitFit (bias-only, \(<0.1\%\) parameters) — yielding 20 (method, architecture) configurations per dataset, evaluated on CIFAR-100 and DTD.

Adaptive gradient checkpointing. At each training step, the algorithm scans layers in forward-pass order and monitors torch.cuda.memory_allocated(). As soon as allocated memory exceeds \(\tau \cdot M\) (where \(M\) is the VRAM budget and \(\tau \in (0,1]\)), a flag is set and all subsequent layers in that step are checkpointed (activations discarded, recomputed during backprop); earlier layers retain activations normally. The cascade from layer \(i^*\) is justified by the non-decreasing property of cumulative activation memory during a single forward pass. \(\tau\) is calibrated via a descending sweep from 0.9 in steps of 0.1, accepting the largest \(\tau\) for which 5 training steps complete OOM-free with peak memory \(\leq 0.9 M\); across all four architectures at 2 GB, this converged to \(\tau = 0.5\). The flag resets each step, allowing re-adaptation as optimizer-state allocations change.

Foundation-model baselines. Three families are evaluated zero-shot/training-free: contrastive VLMs (OpenCLIP ViT-L/14, SigLIP-Large), KD-SSL backbone (DINOv2-Large with \(k\)-NN or linear probe), and autoregressive VLMs (SmolVLM, MobileVLM v2, PaliGemma-3B).

Metrics. All configurations are compared on top-1 accuracy, training time, energy (Wh), peak VRAM, and the NetScore family. The paper introduces two new deployment-aware NetScore variants: \(\text{NS}_M\) (accuracy–memory trade-off) and \(\text{NS}^\#\) (accuracy–memory–energy trade-off), extending the original NS and \(\text{NS}_E\) formulations.

Key Contributions

  • Systematic evaluation of five PEFT methods on four Transformer and Mamba backbones under a fixed 2 GB VRAM budget, with quantitative multi-objective ranking.
  • Adaptive gradient checkpointing algorithm that monitors runtime GPU memory and checkpoints only layers exceeding \(\tau \cdot M\), reducing peak memory 43–79% with lower energy overhead than static checkpointing.
  • Two new NetScore variants (\(\text{NS}_M\), \(\text{NS}^\#\)) that add memory and energy axes to the accuracy–efficiency metric family.
  • Cross-paradigm comparison of PEFT fine-tuning against contrastive VLMs, KD-SSL (DINOv2), and autoregressive VLMs on matched datasets and energy budgets.

Results

  • QLoRA and BitFit reduce energy 20–30% relative to Full FT at only 1–2% accuracy cost.
  • Adaptive checkpointing reduces peak VRAM by 43–79% compared to no checkpointing, with 9–30% energy overhead (compared to the larger overhead of static per-layer checkpointing).
  • DINOv2 (zero-shot, linear probe) achieves 0.917 top-1 accuracy on CIFAR-100, surpassing the best fine-tuned model (0.897) while consuming a fraction of the energy.
  • Small autoregressive VLMs (SmolVLM, MobileVLM v2, PaliGemma-3B) remain uncompetitive with fine-tuned models and DINOv2 on both accuracy and energy.
  • Static checkpointing imposes 17–44% wall-clock energy overhead on ViT-S and TinyViT across both datasets, consistent with the analytical \(\approx 33\%\) step-cost bound.
  • TinyViT Full-FT with adaptive checkpointing at \(\tau = 0.5\) triggers OOM at batch size 32 due to the MBConv stem's first-layer activation footprint; it requires batch size 16 (peak 1,760 MiB).

Limitations

  • Only two datasets (CIFAR-100, DTD) are used; generalization to other domains (medical, satellite, fine-grained) is unverified.
  • \(\tau\) calibration is empirical (descending linear sweep with step 0.1); finer-grained sweeps (step 0.05) or principled priors are left as future work.
  • Adaptive checkpointing assumes a near-uniform per-layer activation footprint; it fails or underperforms static checkpointing on heterogeneous hybrids like TinyViT (MBConv + attention), which requires a stage-aware threshold \(\tau_s\).
  • Only classification tasks are evaluated; detection, segmentation, and generation tasks are out of scope.
  • Autoregressive VLM baselines are used only for zero-shot prompt-based classification; fine-tuned VLM performance (e.g., LoRA on PaliGemma for the same task) is not compared.
  • Energy measurements are hardware-specific (GTX 1650 / Jetson Nano class); results may not transfer to other GPU families.

Relevance to Vision-Language Models

This paper directly benchmarks contrastive VLMs (OpenCLIP, SigLIP) and autoregressive VLMs (PaliGemma, MobileVLM, SmolVLM) against PEFT fine-tuned vision backbones under realistic memory and energy constraints, providing concrete evidence that small autoregressive VLMs are currently uncompetitive for standard classification tasks while contrastive and KD-SSL models (DINOv2) can outperform fine-tuning entirely. The methodology and new \(\text{NS}_M\)/\(\text{NS}^\#\) metrics are directly applicable to evaluating PEFT strategies for VLMs themselves (e.g., LoRA on vision encoders or cross-modal adapters) under edge deployment constraints. The finding that activation memory — not parameter count — dominates VRAM during PEFT is particularly relevant for VLM researchers designing memory-efficient fine-tuning pipelines for large vision encoders paired with language decoders. The adaptive checkpointing algorithm is architecture-agnostic and could be applied to the vision encoder or the full VLM fine-tuning stack.