RADIO1D: Elastic Representations for Condensed Vision Modeling¶

🕒 Published (v1): 2026-07-03 22:58 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML; challenges fixed 2D patch assumption in VLMs with elastic 1D representations

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

RADIO1D replaces fixed 2D patch token sequences in vision encoders with a variable-length 1D token representation learned via multi-teacher knowledge distillation and nested dropout. A pre-analysis shows VLM fine-tuning already destroys spatial coherence in 2D encoders (CKA$_{\text{off}}$ drops from 0.281 to 0.035), motivating a representation explicitly designed around global summarization. The resulting encoder serves a single VLM checkpoint across arbitrary token budgets at inference time, achieving Pareto-optimal accuracy–latency tradeoffs on 10 multimodal benchmarks.

Problem¶

Standard VLM vision backbones produce fixed-length 2D patch token sequences. Natural images exhibit strong spatial correlations across patches ($I(Z_i; Z_j | X) > 0$), causing redundancy that bloats $H(Z)$ and degrades the rate–distortion tradeoff under token-count constraints. Moreover, empirical analysis shows VLM fine-tuning discards the very spatial coherence that 2D encoders are designed to preserve, making fixed-grid representations a poor architectural fit for the actual optimization target of cross-modal alignment.

Method¶

RADIO1D is an encoder–decoder architecture initialized from C-RADIOv4-H and fine-tuned with agglomerative multi-teacher distillation (teachers: SigLIP2-g, DINOv3-7B, SAM3) on a royalty-free DataComp1B subset for 300k steps.

Encoder operates in strict 1D mode: the input image is patchified into a standard 2D grid, flattened, and processed through a ViT with Patch Merging blocks (2×2 pixel unshuffle → linear projection to $\rho d$ channels, $\rho=2$) inserted at backbone depth $\kappa=24$ to reduce sequence length. The encoder outputs a variable-length sequence of $\ell$ tokens, where $\ell$ is drawn from a triangular prior $p(x) = 2 - 2x$ during training (nested dropout): earlier tokens are always retained and learn global semantics; later tokens encode finer details.

Decoder (training only) reconstructs a 2D-compatible patch grid from the $\ell$ 1D tokens via cross-attention: learnable query tokens (matching teacher spatial resolution) attend to the 1D tokens as register keys/values. This 2D output is fed through teacher-specific MLP adapters and aligned to each teacher via per-teacher losses: $$\mathcal{L}_t = \ell_s(\hat{z}^s_{(t)}, z^s_{(t)}) + \frac{1}{N}\sum_{i=1}^N \ell_p(\hat{z}^p_{(t,i)}, z^p_{(t,i)}), \qquad \mathcal{L} = \sum_t \lambda_t \mathcal{L}_t$$

At inference, only the encoder runs, emitting $\ell$ tokens (user-specified).

Key Contributions¶

Feature analysis: CKA matrices on MS-COCO show VLM fine-tuning collapses spatially coherent C-RADIOv4 features (DINOv3-like, CKA$_\text{off}=0.281$) into diffuse, SigLIP2-like representations (CKA$_\text{off}=0.035$), empirically validating that spatial coherence is not what VLMs retain.
Global token discovery: Per-token k-NN ImageNet accuracy and CKA analysis identify three anomalous implicit global summary tokens in SigLIP2-SO400m (indices 13, 176, 242) that disproportionately encode whole-image semantics.
Elastic 1D encoder: Nested dropout with triangular length sampling induces strict hierarchical ordering: $I(T_i; Y) > I(T_j; Y)\ \forall i < j$, empirically verified via rate–distortion curves.
Composition-aware retrieval metric: A new evaluation criterion measuring both object presence and spatial arrangement for image retrieval, where compact RADIO1D tokens outperform standard encoders.
Single-checkpoint flexible VLM deployment: One RADIO1D checkpoint supports 1–256 tokens at inference, enabling continuous accuracy–latency tradeoffs without retraining.
Permissive model release: Checkpoints released under an open license.

Results¶

VLM average accuracy (10 benchmarks, Nemotron-Nano-9B-v2 + 17M training samples):
RADIO1D at 256 tokens: 73.3% vs. SigLIP2-g: ~73.1%, C-RADIOv4-H: ~72.8%, SigLIP2-SO400m: ~72.7%
RADIO1D at 224 tokens: 73.0%; 128 tokens: 71.6%; 64 tokens: 70.1%; 32 tokens: 68.1%; 8 tokens: 63.9%; 1 token: 51.5%
All RADIO1D points from a single VE checkpoint; baselines use fixed 256-token output (1024 spatial → 2×2 unshuffle)
RADIO1D is Pareto-optimal on the accuracy vs. time-to-first-token frontier
ADE20K mIoU (semantic segmentation, linear probe):
RADIO1D first token: 40.23 mIoU (vs. 8.05 for last token alone; 5× gap)
First 16 tokens: 54.11 mIoU vs. last 16 tokens: 20.84 mIoU
Full token count (256): 55.83 mIoU
Rate–distortion: RADIO1D achieves strictly lower relative distortion $D = (mIoU_{\max} - mIoU)/mIoU_{\max}$ at all rates from 1/256 to 1 compared to DINOv3-H+, C-RADIOv4-H, and SigLIP2-SO400m.
Ablations: Triangular length sampling outperforms uniform across all token counts; downscaling at $\kappa=24$ beats $\kappa \in \{18, 20, 27\}$; pixel unshuffle applied to the 1D output (without learned Patch Merging inside backbone) degrades substantially due to broken spatial locality assumption.

Limitations¶

Extreme token compression causes distinct failure modes on tasks requiring holistic scene understanding (e.g., video reasoning); the performance floor at 1 token (~51.5% VLM accuracy) limits use cases with very tight latency budgets.
Training used half the batch size of C-RADIOv4's corresponding stage; full-scale training may change results.
The decoder is discarded at inference, meaning teacher alignment is indirect—spatial reconstruction quality is never directly supervised at deployment.
Analysis of global token emergence is performed only on SigLIP2-SO400m and DINOv3-H+; generalizability to other encoder families is untested.
Authors are NVIDIA employees, the same organization developing the RADIO model family, introducing potential evaluation bias.

Relevance to Vision-Language Models¶

RADIO1D directly addresses one of the core architectural tensions in VLMs: the mismatch between 2D patch-based vision encoders and the LLM token budget. The empirical finding that VLM fine-tuning destroys spatial coherence (CKA$_\text{off}$ collapse) reframes the encoder design question—summarization capacity may matter more than raw spatial fidelity or image–text pre-alignment. The elastic 1D token sequence enables a new deployment paradigm where a single trained VLM can trade accuracy for latency at inference time by adjusting token count, without architecture changes or retraining, which is practically valuable for serving at variable compute budgets.