Skip to content

Video Generation Models are General-Purpose Vision Learners

🕒 Published (v1): 2026-07-10 01:09 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Video generation as general vision learner; ECCV 2026; Jasper Uijlings (Google); strong foundational claim

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GenCeption repurposes a pre-trained text-to-video diffusion transformer (DiT) as a feed-forward perception backbone, enabling a single unified model to perform dense and sparse vision tasks—depth, normals, segmentation, pose estimation, 3D keypoints, camera pose—steered by text prompts. It matches or exceeds SOTA task-specific specialists while requiring 7×–500× less fine-tuning data. The key thesis is that large-scale video generation constitutes the visual analog of next-token prediction: a universal pre-training objective for generalist vision intelligence.

Problem

Computer vision remains in a "specialized model" era: even the strongest perception foundations (SAM, DepthAnything) are task-specific, require bespoke architectures per task, and lack unified backbones. Prior visual pre-training methods (MAE, DINO, CLIP, VideoMAE) either lack spatio-temporal priors, lack vision-language alignment, or suffer severe scaling limitations compared to language models. No unified, task-agnostic architecture has matched specialist model performance across a broad suite of vision tasks.

Method

GenCeption converts a pre-trained text-to-video Rectified Flow DiT into a single-step feed-forward perception model:

Feed-forward reformulation. The DiT normally predicts velocity \(v = \epsilon - x_0\) under the Rectified Flow objective during iterative denoising. GenCeption instead feeds the clean video latent \(x_0\) as input, fixes the conditioning timestep to \(t = 0\), and uses \(-v = x_0 - \epsilon\) as the output, which empirically aligns better with the target latent and accelerates convergence. Features are harvested from the final DiT layer without architectural modification to the backbone.

Unified task representation. All dense perception outputs (depth, normals, segmentation, DensePose, raymaps) are encoded as 3-channel RGB videos in \([0,1]\), with single-channel outputs replicated and multi-channel outputs (normals, DensePose) mapped to distinct RGB channels. Camera pose is represented as a pixel-space "Rothko" raymap: ray origins in the central region and ray directions in the periphery, fitting 6-channel data into 3 channels. Task selection is entirely data-driven, controlled via the text prompt (e.g., "output: depth").

Sparse task extension. For structured outputs like 2D/3D keypoints, \(T\) learnable tokens (one per frame) are appended to the video latents. After the DiT forward pass, an MLP decodes each token to a \(K\)-dimensional prediction. Temporal position indices are scaled via position interpolation to remain within pre-training bounds.

Synthetic data pipeline. 7,500 human-centric videos are rendered in Blender using 800 RenderPeople assets animated with 200 CMU mocap sequences, with diverse backgrounds, focal lengths, and camera trajectories. Ground-truth depth, normals, segmentation, dense pose, 2D/3D keypoints, and camera poses are produced via dedicated render passes.

Key Contributions

  • Establishes large-scale text-to-video generation as a general-purpose visual pre-training paradigm satisfying spatio-temporal, vision-language, and scalability requirements simultaneously.
  • GenCeption: a single unified DiT backbone + head + loss function architecture for both dense (depth, normal, segmentation, DensePose, raymap) and sparse (2D/3D keypoints) video perception tasks.
  • Feed-forward reformulation of a Rectified Flow DiT: single-pass inference by conditioning on \(t=0\) and negating velocity output, with no backbone modifications.
  • "Rothko" raymap representation for compressing multi-channel camera pose data into 3-channel RGB.
  • Empirical evidence that video generative pre-training outperforms V-JEPA and VideoMAE V2 under identical fine-tuning data budgets.
  • Emergent sim-to-real transfer and out-of-distribution generalization (animals, robots) from a model trained exclusively on synthetic human videos.

Results

  • Depth estimation: matches D4RT and VGGT-Ω with 7×–500Ă— less training data; video generative backbone outperforms the largest V-JEPA and VideoMAE V2 variants at equal fine-tuning data.
  • Multi-task: matches or surpasses DepthAnything V3, SAM3, D4RT, VGGT-Ω, Sapiens, David, Genmo, and Lotus-2 as dedicated specialists across depth, surface normal, camera pose, expression-referring segmentation, and 3D keypoint tasks.
  • Pre-training comparison: video generative backbone > V-JEPA > VideoMAE V2 under comparable settings.
  • Scaling: preliminary data and model scaling properties observed (performance improves monotonically with data and model size).
  • Generalization: model trained on synthetic human data generalizes to real footage and unseen object categories (animals, anthropomorphic characters) without additional training.

Limitations

  • Synthetic training data is human-centric (800 RenderPeople assets, CMU mocap); out-of-distribution generalization to non-human objects is emergent but not systematically trained.
  • Evaluation of scaling properties is described as "preliminary"—no large-scale ablation confirming consistent power-law behavior.
  • Paper is truncated; full quantitative tables and ablations are not available in the provided text.
  • Sparse task extension (keypoints, camera pose) requires appended learnable tokens, which modestly departs from the purely unified architecture goal.
  • Feed-forward single-step formulation trades generative diversity for efficiency; may not capture full uncertainty in ambiguous scenes.
  • Temporal consistency gains over image-based baselines are claimed but detailed video-temporal metrics are not fully reported in the excerpt.

Relevance to Vision-Language Models

GenCeption directly advances the vision-language alignment axis of VLMs by demonstrating that video generative pre-training—natively conditioned on text—produces richer visual representations than image-contrastive methods (CLIP/SigLIP-style) or vision-only SSL for dense perception tasks. For researchers tracking VLMs, this paper proposes a concrete alternative to the standard encode-then-align recipe: use video diffusion as the pre-training substrate so that spatial, temporal, and linguistic grounding are jointly learned from the start. The text-steered, task-agnostic inference paradigm (modality selected via prompt) is also directly relevant to unified VLMs that aim to consolidate perception under language control. The 7×–500× data efficiency result challenges the assumption that dense perception requires large annotated datasets when a strong VL-aligned backbone is available.