Vision Transformers Don't Need Trained Registers¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

High-norm "outlier" tokens in Vision Transformers (ViTs) emerge from a sparse set of MLP neurons ("register neurons") that causally concentrate norm at low-information image patches, creating attention artifacts. This paper identifies these neurons and proposes test-time registers: appending an untrained token and redirecting register-neuron activations into it during inference, no retraining required. The method matches the downstream performance of models explicitly retrained with learned registers across classification, segmentation, depth estimation, and unsupervised object discovery.

Problem¶

Darcet et al. (2024) showed that ViTs (e.g., DINOv2, CLIP) develop high-norm outlier patch tokens that pollute attention maps and degrade dense prediction tasks. Their fix—adding learned "register" tokens during pretraining—requires retraining from scratch and is therefore unavailable for released checkpoints. No training-free post-hoc remedy existed.

Method¶

The authors first localize the mechanism: tracking per-layer max norms on 1000 ImageNet images reveals that outliers appear after the MLP of a specific layer (layer 6 in OpenCLIP ViT-B/16). A neuron-level sweep identifies a small set (~5–10 out of ~100K) of MLP neurons whose average activation is consistently highest at outlier patch positions across images—these are "register neurons." Detection is formalized in Algorithm 1: for each candidate neuron in layers up to top_layer, average its activation over identified outlier positions across an image set; return the top_k highest-scoring neurons.

To create test-time registers: append one zero-initialized extra token to the input sequence. During the forward pass, at each register neuron's layer, copy that neuron's maximum patch activation into the extra token's position and zero out its activation everywhere else, then resume normal computation. This redirects the high-norm activity out of image patches and into the dedicated token without any weight updates. A complementary mathematical model (Proposition 1 / Corollary 1) shows register neurons satisfy the conditions for inducing attention sinks under the no-op attention framework, and that their aggregate effect is equivalent to injecting learned key-value attention biases.

Key Contributions¶

Identification of "register neurons"—a sparse, consistent set of MLP hidden units causally responsible for creating high-norm outlier tokens in ViTs (OpenCLIP ViT-B/16 and DINOv2-L/14).
Demonstration that intervening on register-neuron activations shifts outlier positions to arbitrary tokens, including out-of-image tokens.
A training-free test-time register algorithm that appends an untrained token and redirects register neurons into it, matching retrained-register performance.
A formal mathematical model proving register neurons induce attention sinks and implicit attention biases under the no-op framework (Proposition 1, Corollary 1), validated empirically.
Application to VLMs: test-time registers applied to the CLIP vision encoder of LLaVA-Llama-3-8B remove outlier artifacts from language-model cross-attention to visual tokens, improving interpretability while preserving benchmark scores.

Results¶

Linear probing (DINOv2 ViT-L/14): ImageNet 86.4% (base) → 86.4% (test-time) vs. 86.7% (trained registers); ADE20k mIoU 48.3 → 49.1 vs. 49.1; NYUv2 RMSE 0.388 → 0.378 vs. 0.382.
Zero-shot ImageNet (OpenCLIP): ViT-L/14 76.4% → 76.4%; ViT-B/16 71.3% → 71.3% (no degradation).
Zero-shot segmentation (ImageNet-Seg, DINOv2 ViT-L/14): mIoU 38.3 → 38.9 (trained registers: 33.9); mAP 80.6 → 81.1 (trained: 79.0).
Unsupervised object discovery / LOST (DINOv2 ViT-L/14): corloc VOC07 32.2 → 53.8 (+21.6) vs. trained registers 56.2; VOC12 36.8 → 57.9 vs. 60.2; COCO 25.4 → 41.9 vs. 42.3.
VLM benchmarks (LLaVA-Llama-3-8B, 8 tasks): average 46.2 → 46.2 (no loss); individual scores fluctuate by ≤0.5 points.
Ablation: zeroing register neurons (instead of redirecting) drops OpenCLIP ViT-B/16 zero-shot accuracy from 70.4% to 55.6%; zeroing 10 random neurons causes only 69.3% ± 1.1%.

Limitations¶

Analysis focuses on MLP neurons only; attention layers and positional encodings may also contribute to high-norm tokens (acknowledged by authors).
Test-time registers are not fully equivalent to trained registers: linear-probing Table 1 shows small gaps (e.g., 84.5% vs. 85.6% on ImageNet for the test-time register token).
Evaluation is largely restricted to OpenCLIP and DINOv2; generalization to other pretrained ViT families is untested.
OpenCLIP has no publicly available trained-register checkpoint, so direct comparison is impossible for that model family.
The benefit for OpenCLIP on unsupervised object discovery is marginal, consistent with the observation that trained registers also help little there.

Relevance to Vision-Language Models¶

CLIP-based vision encoders power most VLMs, and the outlier-token artifact directly corrupts the language model's cross-attention to visual tokens, producing misleading text-to-image attribution. This paper provides a plug-in, zero-cost fix: applying test-time registers to the CLIP encoder of LLaVA-Llama-3-8B eliminates those artifacts without touching the LLM weights or retraining the vision encoder. For researchers building or probing VLMs, it also offers a cleaner lens on what visual information the language model actually attends to, which matters for hallucination analysis and mechanistic interpretability. More broadly, the finding that a handful of neurons in a shared vision encoder mediate a global computational role (outlier absorption) rather than feature detection raises important questions about how VLMs route and exploit visual information internally.