HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better

🕒 Published (v1): 2026-07-06 10:06 UTC · Source: Arxiv · link

Why this paper was selected

Tencent HunyuanOCR-1.5; lightweight OCR-VLM unifying doc parsing, extraction, translation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HunyuanOCR-1.5 is a lightweight, end-to-end OCR-specialized VLM that upgrades HunyuanOCR-1.0 along two axes: inference speed and capability coverage. It applies DFlash speculative decoding (block-diffusion draft model) to achieve 6.37× Transformer and 2.14× vLLM speedups, while an agent-driven data construction system called Agentic Data Flow closes long-tail capability gaps across ancient scripts, low-resource languages, and multi-image QA. The model claims top performance on OmniDocBench v1.6 among lightweight end-to-end OCR VLMs.

Problem

Existing OCR-specialized VLMs face two deployment barriers: (1) long autoregressive decoding is a latency bottleneck for dense structured outputs (tables, formulas, multi-column documents), and (2) capability coverage is narrow—models focus on document parsing but underperform on ancient-script OCR, low-resource multilingual parsing, multi-image document understanding, and hallucination avoidance. General-purpose VLMs (GPT-4o, InternVL, Qwen-VL) handle OCR tasks but require large model sizes and are not optimized for high-resolution or production-scale OCR workloads.

Method

Architecture: Inherits HunyuanOCR-1.0's compact pipeline—Hunyuan-ViT visual encoder (extended to 4K resolution), adaptive MLP connector, and 0.5B Hunyuan language model with XDRoPE. No backbone redesign.

DFlash inference acceleration: Adapts block-diffusion speculative decoding to OCR decoding. A ~90.7M-parameter draft model (initialized from the last 5 decoder layers of the target model, 5-layer Transformer) proposes a block of \(B=16\) candidate tokens in one parallel forward pass conditioned on the target model's cached hidden states. The target model then verifies the block and accepts the longest valid prefix, preserving the output distribution. Training freezes the target model and optimizes only the draft model via a position-weighted cross-entropy loss:

\[\mathcal{L}_{\text{DFlash}} = \frac{1}{Z} \sum_{j=1}^{n} \sum_{k=1}^{B-1} w_k^{(j)} \left[ -\log p_\theta\!\left(y_k^{(j)} \mid h_{<a_j}, m_{1:B}\right) \right]\]

with exponential decay weights \(w_k^{(j)} = \mathbb{1}[k>0] \cdot \mathbb{1}[\text{valid}] \cdot \exp(-\max(k-1,0)/\gamma)\), \(\gamma=7.0\), \(n=16\) anchors per sequence, trained with a FlexAttention block-diagonal mask.

Agentic Data Flow: An LLM agent equipped with tool-calling (web search, OCR services, VLM APIs, file-processing scripts) accepts capability-gap descriptions in natural language, decomposes them into data requirements, autonomously collects and verifies materials, and iterates data pipelines in a closed loop with algorithm engineers. Instantiated for: (1) low-resource OCR—synthetic multilingual rendering across 331 languages using collected TTF fonts and corpora; (2) ancient-script OCR—seven historical Chinese character forms with layout-aware synthesis; (3) multi-image QA—cross-page QA generation from multi-page PDFs using a strong text model.

Training: Pretraining Stage 3 re-planned to include Agentic Data Flow outputs, multi-image data, and historical OCR data at 4K resolution and 128K context. Post-training refines SFT data and applies RL exploration across OCR tasks.

Key Contributions

• DFlash speculative decoding adapted to OCR VLM inference, achieving 6.37× speedup (Transformers) and 2.14× speedup (vLLM) for long structured outputs while preserving output distribution.
• Agentic Data Flow: an agent-driven data construction system that converts identified model weaknesses into executable, iterable data pipelines without exclusively manual scripting.
• Pretraining extended to 4K input resolution and 128K context window; post-training with task-specific RL.
• SOTA on OmniDocBench v1.6 among lightweight end-to-end OCR VLMs.
• PC-side deployment support via llama.cpp (CPU, consumer GPU, laptop).
• Open release of model weights and training code.

Results

• Inference speed (DFlash): 6.37× speedup over standard AR decoding under Transformers; 2.14× under vLLM. Acceleration ratio increases with output length and is most pronounced for highly structured outputs (HTML tables, formulas, structured Markdown).
• OmniDocBench v1.6: Top-tier performance among all lightweight end-to-end OCR VLMs (specific numbers not provided in the supplied text).
• Low-resource multilingual OCR: 331-language coverage via synthetic data pipeline.
• Ancient-script OCR: Coverage of seven historical Chinese character forms.
• Baselines implicitly include HunyuanOCR-1.0, general VLMs (GPT-4o, InternVL, Qwen-VL), and other OCR-specialized VLMs (modular and end-to-end); head-to-head tables not included in the supplied excerpt.

Limitations

• Full quantitative benchmark tables (OmniDocBench, text spotting, translation, hallucination) are not included in the provided text excerpt, making independent numerical verification impossible.
• DFlash introduces a separate ~90.7M-parameter draft model that must be trained and co-deployed, adding engineering complexity.
• DFlash speedup is most effective at low concurrency (memory-bandwidth-bound regime); high-concurrency serving with vLLM yields a lower 2.14× gain, suggesting diminishing returns under heavy load.
• Agentic Data Flow relies on a closed-loop with human algorithm engineers for quality control; it is not fully autonomous.
• Multi-image QA filtering (removing single-page-answerable questions) is applied but not fully described in the provided excerpt.

Relevance to Vision-Language Models

HunyuanOCR-1.5 is directly relevant to researchers studying how to specialize VLMs for high-density text-centric perception without scaling up model parameters—demonstrating that targeted data construction and inference-time acceleration can move the capability frontier more efficiently than architecture redesign. The DFlash integration shows a concrete path for applying block-diffusion speculative decoding to structured-output VLMs, which generalizes beyond OCR to any VLM generating long, locally regular token sequences (code, tables, LaTeX). The Agentic Data Flow paradigm is notable as a systematic method for closing long-tail coverage gaps in VLMs through agent-driven data synthesis, relevant to the broader challenge of capability extension in specialized VLMs. Together, these contributions address a persistent tension in the VLM field between model compactness, inference efficiency, and task breadth.