Skip to content

PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Positional information preserved during token merging prevents spatial degradation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PPE (Positional Preservation Embedding) is a parameter-free, plug-and-play positional encoding operator that preserves spatiotemporal layout during visual token merging in MLLMs. By partitioning RoPE embedding dimensions into \(K\) chunks and storing \(K\) distinct position IDs per merged token, each compressed token encodes multiple original positions. Applied on top of Chat-UniVi, PPE achieves consistent 2–5% gains across image and video benchmarks at up to 94% token reduction.

Problem

Existing clustering-based visual token compression methods (e.g., Chat-UniVi, PACT) reduce sequence length but discard or poorly preserve spatial and temporal positional information. Chat-UniVi assigns random position IDs to cluster representatives; PACT retains only the cluster center's ID. Both approaches corrupt fine-grained layout cues needed for tasks like text recognition, document understanding, and temporal grounding.

Method

PPE exploits the dimension-independence of RoPE: for a token vector \(\mathbf{z} \in \mathbb{R}^D\), standard RoPE applies \(\text{RoPE}(z_d, m) = e^{im\theta_d} z_d\) with a single position \(m\). PPE instead partitions the \(D\) dimensions into \(K\) equal chunks and assigns a different position ID \(m_k\) to each chunk:

\[\hat{m}_d = m_{k,d}, \quad d = (k-1)\tfrac{D}{K}+1 \;\ldots\; k\tfrac{D}{K}.\]

For M-RoPE (3D spatiotemporal), \(K\) is set to the GCD of the mrope section sizes so each spatial/temporal axis is evenly partitioned. The \(K\) IDs per cluster are selected as the top-\(K\) tokens closest to the cluster center (highest importance score). This is applied before and/or within LLM layers for cascade compression: at each compression stage, already-merged tokens carry their accumulated position sets, and repeated merges reduce the effective set size naturally.

Key Contributions

  • Identifies that position ID treatment (random, center-only, or multi-ID) is the critical axis differentiating token compression quality.
  • Proposes PPE: encodes \(K\) original position IDs into a single merged token via dimension-splitting of RoPE/M-RoPE, with zero learnable parameters and zero added FLOPs.
  • Extends PPE to cascade (multi-stage, within-LLM-layer) compression, enabling ~90% token reduction without premature semantic collapse.
  • Demonstrates that the same plug-in works for both 2D (image, \(K=32\)) and 3D spatiotemporal (video, \(K=8\)) M-RoPE settings.

Results

  • Image (55% token reduction): PPE (79.32% avg) vs. Chat-UniVi (67.61%) — \(+11.71\%\); TextVQA: 77.14 vs. 57.66 (\(+19.48\)); DocVQA: 76.79 vs. 52.48 (\(+24.31\)); OCRBench: 691 vs. 535.
  • Video, spatial-only (55% reduction): PPE 58.69% avg vs. Chat-UniVi 56.74% (\(+1.95\%\)); VideoMME w/o subs: 58.70 vs. 57.22; NeXT-QA OE: 32.61 vs. 25.37.
  • Video, spatiotemporal (94% reduction): PPE 57.62% avg vs. Chat-UniVi 55.70% (\(+1.92\%\)).
  • Cascade compression (90% within-LLM): VideoMME 58.48 (within-LLM PPE) vs. 57.41 (before-LLM only at same 90% budget).
  • Vs. dense models at 55% reduction: PPE-3B (58.48 VideoMME, 84.78 MMBench EN) is competitive with Qwen2.5-VL-3B dense (61.50, 79.10) and PACT-7B (57.60, 80.30).
  • At 90% reduction: PPE*-3B maintains comparable VideoMME performance to the 55%-reduction variant.

Limitations

  • Evaluated on a single base model (Qwen2.5-VL-3B-Instruct); generalization to other architectures is deferred to ablations in appendix.
  • \(K\) (capacity) is a fixed hyperparameter tied to the GCD of M-RoPE section sizes, coupling PPE design to the model's positional encoding layout.
  • Training uses downsampled datasets (120K video / 300K image samples), so absolute scores may not reflect full-scale fine-tuning.
  • During repeated cascade merges, the number of retained position IDs decreases, potentially losing positional fidelity at very deep compression stages.
  • No analysis of wall-clock inference speedup beyond token count reduction ratios.

Relevance to Vision-Language Models

PPE directly addresses a core efficiency bottleneck in MLLMs — redundant visual tokens — while recovering the spatial and temporal structure that layout-sensitive VL tasks require. The approach is particularly relevant to the trajectory of high-resolution and long-video MLLMs (LLaVA-OneVision, Qwen2.5-VL, Video-LLaVA) where token counts scale quadratically with resolution or frame count. The idea of encoding multi-position information per token via RoPE dimension splitting is a lightweight but principled alternative to architectural changes like cross-attention pooling or Q-Former adapters. For researchers tracking VLM efficiency and spatiotemporal reasoning, PPE provides a clean ablation showing that positional ID treatment — not just cluster quality — is a first-order factor in compression fidelity.