Skip to content

PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PerceptionLM (PLM) is a fully open, reproducible vision-language model trained without distillation from proprietary models, designed to enable transparent VLM research. The authors identify synthetic data as insufficient for fine-grained video understanding and fill the gap with 2.8M human-labeled fine-grained video QA pairs and spatio-temporal captions. They also introduce PLM–VideoBench, a benchmark targeting "what/where/when/how" video reasoning that existing benchmarks neglect.

Problem

State-of-the-art VLMs increasingly rely on distillation from closed proprietary models (GPT-4o, Gemini) to label training data, which (1) makes scientific progress unmeasurable because benchmark gains may reflect teacher model contamination rather than algorithmic advances, and (2) leaves fundamental questions about VLM training from scratch unanswered. Additionally, existing human-annotated video datasets are small-scale and lack fine-grained spatio-temporal coverage, and no benchmark adequately tests these capabilities.

Method

PLM pairs a pre-trained Perception Encoder (PE-L/14 at 300M or PE-G/14 at 2B) with a Llama 3 language decoder (1B/3B/8B) connected via a 2-layer MLP projector. Training follows three stages:

  1. Projector warm-up: freeze encoder and LLM, train only the projector on 1M SA-1B synthetic image captions.
  2. Large-scale midtraining: train on 72M diverse synthetic image+video samples (up to 16 tiles/16 frames) generated by an open-source-only data engine using Llama 3 for captioning and QA generation.
  3. Supervised fine-tuning: train at higher resolution (36 tiles/32 frames) on 19M samples including the new human-annotated data.

The open-source data engine generates ~66.1M samples across natural images, charts/documents, and egocentric/exocentric videos entirely without proprietary models. Scaling law experiments (power-law fit Err = (β × FLOP)^α) reveal synthetic data scales well for generic tasks (α = −0.15 for VideoQA) but poorly for hard fine-grained tasks (α = −0.03). To address this, they collect: - PLM–FGQA: 2.4M human-verified QA pairs across 780K video clips covering action/object recognition, motion direction, counting, object state, spatial relations, and pose—~8× larger than any prior human-annotated video QA dataset. - PLM–STC: 194.2K spatio-temporal captions associating each caption with a timestamp interval and a SAM-2-generated masklet (subject segmentation), yielding three derived tasks (RCap, RTLoc, RDCap).

PLM–VideoBench compiles five evaluation tasks from these annotations: FGQA (MCQ, MBAcc metric), SGQA (open-ended egocentric QA, LLM-judge), RCap (region captioning, LLM-judge), RTLoc (temporal localization, recall@1 over IoU 0.3–0.9), and RDCap (dense captioning, SODA score).

Key Contributions

  • Fully reproducible VLM (PLM, 1B/3B/8B) trained exclusively on open-access data and open-source synthetic pipelines—no proprietary model distillation.
  • Open-source data engine generating 66.1M synthetic image/video samples with empirical scaling laws characterizing its limits.
  • PLM–FGQA: 2.4M human-labeled fine-grained video QA pairs, ~8× larger than any prior dataset of this type.
  • PLM–STC: 476K spatio-temporal captioning samples (first large-scale dense video-region captioning dataset with per-frame masklets).
  • PLM–VideoBench: Five-task benchmark suite specifically targeting fine-grained spatio-temporal video understanding gaps in existing benchmarks.
  • Identification of the "HardQA" synthetic data scaling failure: challenging video tasks (fine-grained/temporal) show power-law exponent −0.03 vs. −0.15 for generic VideoQA.

Results

  • Image (20 benchmarks, 8B scale): PLM-8B achieves 94.6 ChartQA, 85.5 DocVQA, 86.5 InfoQA, 80.9 MMMU, 92.7 AI2D—comparable to InternVL2.5-8B and Qwen2.5-VL-7B, and greatly outperforms Molmo-7B-O on most tasks.
  • Video (25 benchmarks, 8B scale): PLM-8B scores 84.9 MVBench, 77.1 NExT-QA, 84.1 Video-MME, 84.9 ActivityNet-QA, 68.8 EgoSchema, 61.4 MotionBench, 72.7 TempCompass—competitive with or exceeding Qwen2.5-VL-7B and InternVL2.5-8B.
  • PLM–VideoBench (8B): PLM-8B achieves 55.6 average vs. GPT-4o at 51.6 and InternVL2.5-8B at 38.5. On FGQA, PLM-8B scores 67.7 vs. GPT-4o at 61.2. On spatio-temporal tasks (RDCap, RCap, RTLoc), open-access baselines near-collapse (InternVL2.5-8B: 5.7/26.1/8.8) while PLM-8B scores 52.8/46.6/59.1.
  • Ablation: Adding PLM-STC improves spatial/temporal tasks by +6.8 points; adding PLM-FGQA improves fine-grained video tasks by +1.3 and video hallucination by +3.0; the three-stage pipeline outperforms a merged two-stage setup by +1.2 image / +2.0 video points.

Limitations

  • PLM-8B on SGQA falls 2.1 points below the best open-access model (InternVL2.5) and significantly below GPT-4o, suggesting egocentric zero-shot QA remains a gap.
  • Spatio-temporal tasks (RCap, RTLoc, RDCap) still fall well short of human performance (human avg. 73.9 vs. PLM-8B 55.6), indicating the task is far from solved.
  • Synthetic data scaling is fundamentally insufficient for fine-grained/temporal tasks (α = −0.03), and the human annotation effort required to close this gap is substantial.
  • Model scales are capped at 8B parameters; competitiveness at larger scales (70B+) is not demonstrated.
  • Video input is fixed at 32 frames, which may limit long-video reasoning despite inclusion of long-video benchmarks.

Relevance to Vision-Language Models

PLM directly challenges the prevailing practice of proprietary-model distillation in VLM training by demonstrating that competitive performance across 40 image/video benchmarks is achievable with fully open pipelines—a methodological clarification with broad implications for how the community measures progress. The paper's empirical characterization of synthetic data scaling limits (fine-grained/temporal tasks plateau at α = −0.03) pinpoints exactly where human annotation remains irreplaceable, guiding future data collection efforts for VLMs. The PLM–FGQA and PLM–STC datasets fill a structural gap in video-centric training data for fine-grained activity and spatio-temporal understanding that synthetic generation cannot address. PLM–VideoBench provides a benchmark specifically designed to surface these gaps, useful for evaluating future VLMs on capabilities that existing suites underweight.