Skip to content

HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HoPE addresses the failure of existing multimodal RoPE variants to maintain semantic similarity over long video contexts by theoretically proving that any fixed-frequency temporal allocation eventually breaks semantic preference, then proposing a hybrid strategy that zeros out temporal frequencies (using NoPE-style identity rotation) and adds dynamic bidirectional temporal scaling. On four video benchmarks, HoPE outperforms all RoPE baselines on both long video understanding and retrieval tasks.

Problem

Existing multimodal RoPE extensions (M-RoPE, VideoRoPE) use heuristic frequency allocation strategies to encode 3D spatial-temporal positions in VLMs. The paper proves that regardless of whether highest or lowest frequencies are assigned to the temporal dimension, there exists a critical context length beyond which the semantic preference property (attention should favor semantically similar tokens regardless of distance) is violated. Additionally, existing temporal index scaling is either absent or uses a fixed unidirectional factor, failing to handle real-world variation in video speed and information density.

Method

Hybrid Frequency Allocation (HFA): For a 128-dim RoPE, the first 96 dimensions encode spatial (x, y) with high frequencies in an interleaved pattern. The remaining 32 temporal dimensions are set to frequency θ=0, yielding an identity rotation matrix. This is proven (Theorem 4.1) to maximize the semantic preference guarantee over any alternative frequency choice, because the temporal contribution to the attention score becomes a constant positive term rather than an oscillating cosine.

Dynamic Temporal Scaling (DTS): A set of scaling factors Γ={0.5, 0.75, 1.0, 1.25, 1.5} is defined. During training, γ is sampled randomly per video and applied to visual token temporal indices (Equation 5), enabling the model to learn temporal relationships at multiple scales (both compression γ<1 and expansion γ>1). At inference, γ is chosen task-adaptively: smaller values for retrieval (preserves semantic preference over very long sequences), larger values for understanding (preserves spatial detail).

The model is built on Qwen2-1.5B/7B backbones with vision encoders from Qwen2-VL-2B/7B, trained at 8k context on ~300k video-instruction pairs from LLaVA-Video-178k.

Key Contributions

  • First theoretical analysis proving all fixed-frequency multimodal RoPE strategies violate semantic preference beyond a critical context length (Theorem 3.1).
  • Proof that zeroing temporal frequencies provides the tightest upper bound on semantic preference for any context length (Theorem 4.1).
  • HFA: spatially-interleaved high-frequency + zero-frequency temporal encoding as a principled drop-in RoPE replacement.
  • DTS: bidirectional random temporal scaling during training; task-adaptive scaling at inference.
  • Empirical demonstration of 22.23% improvement on V-NIAH retrieval and 8.35% on long video understanding over the best baseline.

Results

  • Long video retrieval (V-NIAH, 1-hour haystack, ~432k tokens): HoPE surpasses best baseline (VideoRoPE) by +22.23%.
  • Long video understanding (average across MLVU, LongVideoBench, Video-MME): HoPE surpasses best baseline by +8.35%.
  • 7B, MLVU 32k: HoPE 63.85 vs. VideoRoPE 62.51 (+1.34), Vanilla RoPE 61.03 (+2.82).
  • 7B, LongVideoBench 32k: HoPE 55.34 vs. VideoRoPE 53.82 (+1.52), Vanilla RoPE 51.29 (+4.05).
  • 7B, LongVideoBench 64k: HoPE 50.01 vs. VideoRoPE 45.31 (+4.70), Vanilla RoPE 34.38 (+15.63).
  • Ablation (Video-MME 7B): Adding 3D structure alone gives marginal gain; adding HFA improves average across lengths by ~1.69; adding DTS gives further improvement.
  • Benefits scale with backbone size: gains are larger at 7B than 2B.
  • All methods degrade substantially at 64k; HoPE degrades least.

Limitations

  • Experiments are limited to 2B and 7B backbone scales; larger models are not evaluated.
  • Training is capped at 8k context with at most 128 frames; whether HoPE's gains persist with large-scale long-context training data is unexplored.
  • Performance at 64k (8× training length) degrades significantly for all methods, including HoPE.
  • The optimal inference scaling factor γ is task-dependent and requires manual selection; no automatic adaptation is proposed.
  • Theoretical guarantees for HFA assume semantic preference is the sole objective; the tradeoff with spatial detail preservation is noted but not formally characterized.

Relevance to Vision-Language Models

HoPE directly tackles a fundamental bottleneck for long-video VLMs: the inadequacy of positional encodings when sequences extend far beyond pretraining context length. By grounding the design of multimodal RoPE in formal proofs rather than empirical heuristics, it establishes a theoretically principled framework that future VLM position-embedding designs can build on. The zero-frequency temporal strategy connects multimodal RoPE to NoPE research, offering a hybrid that leverages the implicit positional awareness of causal attention for long-range semantics while retaining explicit spatial encoding. This is directly relevant to scaling VLMs to hour-long video understanding—a capability gap that limits deployment in video QA, temporal grounding, and video retrieval.