Skip to content

Uni-MuMER: Unified Multi-Task Fine-Tuning of Vision-Language Model for Handwritten Mathematical Expression Recognition

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Uni-MuMER adapts a general-purpose VLM (Qwen2.5-VL-3B) to Handwritten Mathematical Expression Recognition (HMER) via multi-task fine-tuning—without any architectural changes—using three data-driven auxiliary tasks that address structured spatial reasoning, symbol confusion, and output consistency. It achieves 79.74% average ExpRate on CROHME, outperforming the best specialized model (SSAN) by 16.31% and Gemini2.5-flash zero-shot by 24.42%. With external data (1.6M samples), Uni-MuMER† reaches 82.86% ExpRate@CDM.

Problem

HMER has seen only ~3% absolute improvement over several years (CoMER→SSAN) due to isolated architectural modifications that are hard to unify, single-task training that ignores complementary priors, and poor cross-dataset generalizability. Meanwhile, powerful closed-source VLMs (Gemini2.5-flash) show strong zero-shot HMER capability, but open-source VLMs lag and the recipe for their improvement is unclear.

Method

Uni-MuMER fully fine-tunes Qwen2.5-VL-3B (ViT encoder + transformer decoder) on four jointly trained tasks, all expressed as image-to-text generation:

  1. Vanilla HMER: Direct image → LaTeX mapping; establishes base recognition capability.
  2. Tree-Aware Chain-of-Thought (Tree-CoT): Parses the expression into an Abstract Syntax Tree (AST), depth-first serializes it with tab-indented nodes carrying spatial relation labels (above/below/right/sub/sup), and prepends this linearized tree to the LaTeX output. Forces explicit 2D structural reasoning.
  3. Error-Driven Learning (EDL): Generates an error corpus via out-of-fold cross-validation (train on n−1 folds, multi-sample predictions on held-out fold, compare to ground truth). Trains two subtasks: (a) error detection—mark errors with <error_start>/<error_end>/<deleted> tags; (b) error correction—given marked expression, output correction log and corrected LaTeX. Error corpus scale ≈ original training set size.
  4. Symbol Counting (SC): Prepends a per-symbol count string (e.g., \frac:1,a:1,2:2) to the LaTeX target, compelling the model to explicitly account for every symbol before generating the expression.

All four tasks are trained simultaneously for a single epoch. Uni-MuMER† additionally trains on ~1.6M samples from five datasets (CROHME, CROHME-2023, HME100K, MathWriting, Im2LaTeXv2; ~386K images × 3 tasks + vanilla).

Key Contributions

  • Unified multi-task fine-tuning paradigm for HMER that injects domain knowledge into a VLM without architectural modification.
  • Tree-CoT: novel AST-based chain-of-thought that linearizes 2D expression structure as an intermediate reasoning step.
  • Error-Driven Learning: self-generated error corpus via cross-fold sampling enables targeted correction of visually similar symbol confusions.
  • Symbol Counting auxiliary task to enforce global consistency in long expressions with repeated symbols.
  • New SOTA on CROHME (all years) and HME100K; demonstrated scalability with diverse external data.
  • Faster inference than lightweight specialized models due to compatibility with efficient VLM serving frameworks (vLLM).

Results

  • CROHME Average (ExpRate / ExpRate@CDM):
  • Uni-MuMER (no external data): 73.29% / 77.19% (+4.65pp / +5.37pp over Vanilla baseline)
  • Uni-MuMER†: 79.74% / 82.86% (+11.10pp / +11.04pp over Vanilla)
  • vs. SSAN (best specialized): +16.31pp; vs. Gemini2.5-flash zero-shot: +24.42pp; vs. HiE-VL (VLM-based, external data): +8.61pp on average ExpRate
  • HME100K: Uni-MuMER 71.62% / 73.40%; Uni-MuMER† 72.66% / 74.30% (vs. TAMER 69.50% / 71.30%)
  • Ablation (CROHME-Avg ExpRate): Tree-CoT alone +2.21pp, EDL alone +1.66pp, SC alone +1.22pp; full combination +4.65pp — all tasks complementary.
  • EDL: reduces top-5 alphabet–number confusion count from 5.25 to 3.31 misrecognitions/expression.
  • General capability (Uni-MuMER-LLAVA): mixing 1:1 HMER + LLaVA-OneVision data achieves MMMU_val 48.67%, MathVista 51.1% (vs. Qwen2.5-VL-3B-Instruct: 52.78%, 47.9%), with minimal HMER degradation.

Limitations

  • A 400K synthetic subset of MathWriting was unused due to computational resource constraints, potentially limiting data diversity benefits.
  • Optimal task mixture ratios (Tree-CoT : EDL : SC) and domain mixture ratios (HMER-specific vs. general) were not explored due to resource constraints.
  • Out-of-domain generalization drops markedly when a target domain (e.g., real-world low-quality images in HME100K, printed expressions) is entirely absent from training.

Relevance to Vision-Language Models

This paper demonstrates a clean template for domain-specialized VLM fine-tuning: no architecture changes, only carefully designed auxiliary task data, yet surpassing both task-specific models and large closed-source VLMs. The Tree-CoT technique extends chain-of-thought prompting to 2D structured prediction problems, which is directly relevant to other spatially structured VLM tasks (diagram understanding, table parsing, chemistry structure recognition). The EDL paradigm—using a model's own cross-fold errors as training signal—is a lightweight self-improvement loop applicable broadly to VLM OCR fine-tuning. The result that a 3B-parameter model with proper task design beats Gemini2.5-flash zero-shot on a structured recognition task underscores how far domain-targeted fine-tuning of open-source VLMs can go.