Skip to content

Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

🕒 Published (v1): 2026-05-13 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a systematic study of long-context continued pre-training (LongPT) for vision-language models, identifying practical data-design principles that extend a 7B model from 32K to 128K context. The resulting model, MMProLong, improves long-document VQA by 7.1% over its base and generalizes to 256K–512K contexts without additional training.

Problem

Open-weight LVLMs have rapidly scaled context windows, but practical LongPT recipes remain underexplored: it is unclear which data types to synthesize, how to mix task types, what length distribution to use, and how to preserve short-context capability — technical reports from frontier models (Qwen3-VL, GLM-4.5V) reveal almost none of these details.

Method

Starting from Qwen2.5-VL-7B (native 32K window), the authors run controlled ablations under a fixed 5B-token budget with a 131,072-token max sequence length. They build a 1.5M-document PDF pool and synthesize five training tasks: three long-document VQA types (extract-single, extract-multi, reasoning) and two OCR transcription types (OCR-full, OCR-needle). The VQA pipeline uses a short-to-long strategy: a coherent 8–15 page segment is fed to Seed 2.0 as a QA generator, the resulting QA pair is inserted back into the full-document context, and explicit segment anchors prevent global-context false positives. RoPE is extended via Dynamic-NTK by scaling mRoPE base frequency from \(10^6\) to \(4 \times 10^6\). Three design axes are ablated: (1) sequence-length distribution (pool-native 32–50 page range vs. long-biased, 83.9% ≥100K tokens); (2) extraction-to-reasoning ratio (grid-searched 0:10–10:0); (3) short-context data mixing ratio (0%–80%).

Key Contributions

  • Demonstrates long-document VQA is substantially stronger than OCR transcription for LongPT (best VQA task +5.1–6.3% avg; OCR-full degrades −17.4% without SFT).
  • Shows balanced ("pool-native") length distribution outperforms target-length-focused (long-biased) training, arguing long-context ability requires generalizable retrieval across positions.
  • Finds an 8:2 extraction-to-reasoning mixture is optimal, identifying retrieval as the primary bottleneck in context extension.
  • Shows pure long-document VQA training does not severely degrade short-context performance (short-context avg drop: 66.47 → 65.48), reducing the need for short-data mixing unlike LLM LongPT practice.
  • Releases MMProLong (Qwen2.5-VL-7B backbone), which exceeds all sub-15B open-source models at 128K and generalizes zero-shot to 256K/512K, MM-NIAH needle retrieval, VTCBench compression, and long-video benchmarks.

Results

  • MMProLong vs. Qwen2.5-VL-7B baseline: +7.1% overall average on long-document VQA (MMLongBench, 64K+128K combined: 57.70 vs. 50.59).
  • At 128K, MMProLong scores 55.84 on MMLongBench vs. 48.94 for the base model.
  • Exceeds all small open-source LVLMs (<15B) at both 64K and 128K; matches or exceeds several large open-source models (e.g., Qwen2.5-VL-32B overall avg 58.31 vs. MMProLong 57.70, within noise).
  • At 256K/512K contexts (beyond training window), MMProLong exceeds baselines by over 20%.
  • Pool-native distribution over long-biased: +1.3, +0.1, +1.7 points on the three VQA task types.
  • 8:2 extraction-to-reasoning ratio beats best single-task setting.
  • Recipe validated on Qwen3-VL backbone, confirming transferability.

Limitations

  • Training window capped at 128K; extrapolation to 256K/512K is empirically observed but not architecturally guaranteed.
  • All data sourced from PDFs; generalization to other long-context modalities (e.g., interleaved web pages, code notebooks) not systematically studied.
  • QA generation relies on a proprietary model (Seed 2.0) as the generator; reproducibility depends on access to comparable strong models.
  • Ablations are restricted to a single backbone family (Qwen2.5-VL) until the final cross-backbone validation on Qwen3-VL.
  • Short-context data mixing trade-off is studied only with LLaVA-OneVision data; other instruction-tuning distributions are not explored.

Relevance to Harnesses / Meta-Harnesses

The paper's core contribution is a data-curation and training-recipe harness — a replicable pipeline that orchestrates document pool construction, OCR parsing, segment sampling, QA generation (via an LVLM acting as a generator), instance assembly, and controlled ablations under a fixed compute budget. This is directly analogous to meta-harness design: the short-to-long synthesis pipeline functions as an automated data-generation subagent that must be orchestrated carefully (segment anchoring, quality filtering, length distribution control) to avoid contaminating downstream evaluations. The finding that balanced-length mixtures and retrieval-heavy task compositions outperform naive target-length training informs how harness designers should parameterize automated data pipelines for long-context agentic workflows, where context consistency and key-information retrieval across variable-length histories are the analogous bottleneck.