Perception Encoder: The best visual embeddings are not at the output of the network¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Perception Encoder (PE) is a family of vision encoders built on contrastive vision-language pretraining alone that achieves state-of-the-art on classification, retrieval, MLLMs, and dense prediction tasks. The key insight is that strong general features (for OCR, VQA, detection, depth, tracking) already exist in intermediate layers of a well-trained CLIP model—they just need to be extracted via targeted alignment tuning. Two alignment variants (PE_lang for MLLM tasks, PE_spatial for dense prediction) bring these hidden features to the network's output.

Problem¶

Prior work required multiple specialized pretraining objectives (contrastive for classification/retrieval, captioning for MLLMs, spatially self-supervised for detection/depth) because no single method produced state-of-the-art features for all downstream tasks. Scaling multiple objectives simultaneously is complex. Furthermore, it was believed that CLIP-style contrastive pretraining inherently fails to scale for spatial downstream tasks.

Method¶

PEcore is built in two training stages: (1) robust image contrastive pretraining with progressive resolution (98→448px), large batch sizes (131K), LAMB optimizer, 2D RoPE, attention pooling, data augmentation, and mask regularization; (2) video finetuning using a video data engine that synthesizes aligned captions for 22M clips by combining a PE-based video captioner, a Llama 3.2 frame captioner, and metadata summarized by Llama 3.3 70B—with 120K human-refined captions for quality improvement. Frame embeddings are average-pooled to a single video embedding.

Layerwise frozen feature analysis reveals that intermediate layers of PEcore G already match AIMv2-3B on language tasks and DINOv2-g on spatial tasks. Two alignment methods extract these features:

PE_lang: Unfreezes PEcore (last 3 layers discarded) and jointly finetunes it with a Llama 3.2 3B decoder via next-token prediction on 70M OCR, captioning, VQA, and video QA samples; the encoder is then extracted.
PE_spatial: Uses two teachers—PEcore's own frozen layer 41 features (to preserve high-level semantics) and SAM 2.1 mask logits sampled on a 32×32 grid (to restore spatial locality degraded by global tokens after layer 33). Training uses heavy regularization (DropPath, LayerScale, 75% masking).

Key Contributions¶

Demonstrates that contrastive pretraining alone, with a carefully tuned recipe, learns general features matching specialized pretraining methods at their own tasks—but only in intermediate layers.
Introduces a scalable video data engine combining PE-based captioning, per-frame image captioning (Llama 3.2), and metadata synthesis, bootstrapped with human-refined captions.
PEcore G (1.9B parameters) achieves state-of-the-art zero-shot image and video classification and retrieval, outperforming SigLIP2 and InternVideo2.
PE_lang outperforms all other frozen vision encoders on MLLM benchmarks (OCR QA, VQA, captioning, video QA, grounding) across all scales.
PE_spatial achieves SOTA on frozen dense prediction (DAVIS tracking, ADE20k segmentation, NYU depth) and competitive end-to-end detection (COCO val 66.0 APbox with DETA, matching models trained with specialist pipelines).
Releases models, code, and PE Video Dataset (1M stock footage videos, 120K human-refined captions).

Results¶

Zero-shot image classification: PEcore G (1.9B) scores 85.4 ImageNet val, 80.2 robustness avg, outperforming SigLIP2-g-opt (85.0 / 79.8) and EVA 18B (83.8).
Zero-shot image retrieval: PEcore G avg retrieval 88.2 (MS-COCO), surpassing SigLIP2-g-opt (88.0).
Zero-shot video classification: PEcore G avg 76.9 (Kinetics-400), vs. InternVideo2 73.1 and SigLIP2-g-opt 69.8.
MLLM (PE_lang G, frozen, Llama 3.1-8B decoder): avg VQA 76.2, avg OCR QA 73.6, avg video 58.9—outperforming InternViT2.5-6B (76.8 VQA, 69.0 OCR), SigLIP2-g-opt (77.0 VQA, 55.3 OCR), and AIMv2-3B (75.5 VQA, 38.4 OCR).
Dense prediction (PE_spatial G, frozen): DAVIS tracking 61.5 J&F (vs. DINOv2-g 58.5), ADE20k 49.3 mIoU (vs. DINOv2-g 48.7), NYU depth 0.262 RMSE (vs. DINOv2-g 0.279).
Detection end-to-end: COCO APbox 57.8 (Mask R-CNN), LVIS APbox 54.2; SOTA setting: 66.0 APbox (DETA), matching EVA02-L+CoDETR (65.9) with a simpler decoder.
Video finetuning improves image performance: +0.6% general classification, +1.2% fine-grained, +4.0% retrieval over image-only PEcore G.

Limitations¶

Spatial alignment (PE_spatial) slightly underperforms SAM-aligned models on tracking specifically, due to the compromise between semantic and locality objectives.
The model encodes video as average-pooled frame embeddings with no temporal attention, which limits temporal reasoning despite strong retrieval/classification numbers.
The "best layer" for downstream tasks varies per task; without alignment tuning the last-layer features are substantially worse, requiring task-specific alignment.
General features scale with model size under the robust recipe but the last-layer CLIP embedding stagnates—CLIP loss actively obfuscates the features it learns.
Video data engine quality depends on the image captioner and LLM; errors can compound in the synthesis pipeline.

Relevance to Vision-Language Models¶

PE directly challenges the prevailing assumption that CLIP-style contrastive pretraining is insufficient for tasks requiring rich spatial or language understanding, showing that the right training recipe makes contrastive models competitive with specialized pretraining for MLLMs and dense prediction. The language alignment approach (PE_lang) is directly applicable as a drop-in vision encoder for any MLLM pipeline, outperforming encoders like InternViT-6B at a fraction of the parameters. The finding that intermediate CLIP layers harbor strong task-specific features has implications for how VLM researchers should think about probing, adapter placement, and layer selection in large-scale vision-language models.