Skip to content

Locality Alignment Improves Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Vision-language models inherit spatially-blind ViT backbones (CLIP, SigLIP) trained with image-level supervision, causing systematic spatial reasoning failures. This paper introduces locality alignment, a lightweight post-training stage using a self-supervised procedure called MaskEmbed that forces ViT patch embeddings to encode local (per-patch) semantics by reconstructing masked views of a frozen teacher. Locality-aligned backbones consistently improve VLM performance on spatial benchmarks at under 1% of CLIP/SigLIP pre-training compute.

Problem

Popular ViT backbones (CLIP, SigLIP) used in VLMs are trained with image-level contrastive objectives and weak spatial inductive biases, so their patch embeddings conflate information across locations rather than encoding what is where. Because most VLM recipes freeze or only lightly fine-tune the ViT, this failure mode propagates directly into downstream spatial understanding deficits (object localization, counting, relational QA). Existing alternatives—segmentation datasets (scarce), dense self-supervised pretraining (MAE/BEiT, low semantic richness), or human-annotated region-text pairs—are either expensive or insufficient.

Method

Locality alignment is a post-training stage applied to a frozen-teacher ViT before VLM fine-tuning:

  1. Key insight — masking reveals local semantics: A pre-trained model with good global representations changes its output predictably when patches are masked; comparing outputs under different masks reveals each patch's semantic contribution.

  2. MaskEmbed procedure: An encoder (initialized from the pre-trained ViT) and a lightweight two-layer transformer decoder are trained jointly. For a random binary mask m, the encoder receives masked patch embeddings (masked positions zeroed); the decoder predicts the frozen teacher's full embedding sequence f(m(x)) under the same mask. Loss: MSE between decoder output and teacher's masked embedding sequence (second-to-last layer), not just the [CLS] token.
    Formally: min_{θ,ϕ} E_{x,m}[‖h_ϕ(m(g_θ(x))) − f(m(x))‖²]

  3. Three masks per batch: Each iteration uses mask m, its complement 1−m, and the identity mask (all patches visible), tripling signal per image.

  4. Training: ImageNet-21k, 5 epochs (~60k gradient steps, batch 1024), a two-layer transformer decoder—total cost <1% of CLIP/SigLIP pre-training.

  5. VLM integration: The MaskEmbed decoder (not a new MLP adapter) is repurposed as the vision-language adapter, bridging the locality-aligned embedding space to the LM token space. The ViT is frozen during VLM training (Prismatic/LLaMA-2 7B recipe).

Key Contributions

  • MaskEmbed: Self-supervised masked reconstruction fine-tuning for ViTs using the pre-trained model itself as teacher; requires no human annotations and no new data.
  • Locality alignment stage: Generalizable post-training procedure applicable to any image-level–supervised ViT (CLIP, SigLIP, MoCo, OpenCLIP, DFN, EVA02).
  • Patch-level probing benchmark: MSCOCO cast as patch-wise multi-label classification with frozen backbone + small transformer head, separately measuring local vs. global probing recall.
  • Consistent VLM improvements: Controlled comparisons (same Prismatic recipe, same data) show gains across spatial benchmarks with both CLIP ViT-L @ 336px and SigLIP SO400M @ 384px.
  • Decoder-as-adapter insight: Reusing the MaskEmbed decoder as the VLM adapter is critical; an MLP adapter on locality-aligned embeddings slightly hurts performance.

Results

  • Probing benchmark (MSCOCO patch-level multi-label recall):
  • CLIP ViT-L @ 336px: local recall improves substantially (qualitative Figure 3 shows corrected object localization)
  • SigLIP SO400M @ 384px: similarly strong gains; locality alignment also improves global probing for most models
  • MaskEmbed vs. CLIPSelf (CLIP ViT-B): CLIPSelf degrades local probing from 44.63 → 36.16; MaskEmbed (3× augments) reaches 46.32
  • MaskEmbed with averaged embeddings (no learned decoder): 40.97 — confirms decoder is critical
  • VLM benchmarks (Prismatic + LLaMA-2 7B, Llava-1.5 and extended data mixtures):
  • Spatial understanding gains: RefCOCO, RefCOCO+, RefCOCOg, OCID-Ref, TallyQA (simple/complex), VSR, AI2D — improvements across both CLIP and SigLIP backbones in nearly all conditions
  • Holistic QA (VQAv2, VizWiz) and hallucination (POPE): broadly maintained or improved
  • Feature fusion with DINOv2 can yield larger gains on RefCOCO/TallyQA but degrades TextVQA and VizWiz; locality alignment does not cause such regressions
  • Compute: <1% of CLIP/SigLIP pre-training cost for the full locality alignment stage

Limitations

  • Evaluated on a single VLM architecture (Llava-style patches-as-tokens, Prismatic recipe, ViT frozen during VLM training); benefits may differ with end-to-end fine-tuning or other architectures.
  • Only tested with LLaMA-2 7B; scaling behavior with larger LMs is unknown.
  • Training data for MaskEmbed (IN1k/IN21k) may be less diverse than web-scale CLIP pre-training data, potentially limiting localization of text-heavy or rare visual concepts.
  • MaskEmbed degrades CLIP's zero-shot classification ability (patch embeddings live in a different space than text-aligned CLIP features).
  • No comparison with LocCa due to unavailability of public checkpoints.
  • Does not explore composition with multi-crop features, increased resolution, or prefix attention—these interactions are left to future work.

Relevance to Vision-Language Models

This paper directly targets a known weak point in standard VLM recipes: the spatial blindness of CLIP/SigLIP backbones, which are the de facto choice for most open-source and frontier VLMs. The key contribution—a cheap, annotation-free post-training stage that improves local feature extraction without sacrificing global semantics—slotted directly into existing pipelines (Prismatic, LLaVA-style) demonstrates practical transferability. The insight that the MaskEmbed decoder serves better as a VLM adapter than a standard MLP is a non-obvious finding with immediate architectural implications. For those tracking VLMs, this work is complementary to approaches like multi-crop ensembling, DINOv2 fusion, and high-resolution encoding, and suggests locality alignment as a composable building block to resolve spatial failures without heavy re-pretraining.