Skip to content

TABLET: A Large-Scale Dataset for Robust Visual Table Understanding

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Mirella Lapata; large-scale real-world dataset for robust visual table understanding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TABLET is a large-scale visual table understanding (VTU) dataset of 4 million instruction-formatted examples grounded in 2 million unique tables, 88% of which preserve original (lossless) visualizations rather than synthetic re-renderings. It also introduces VisualTableQA, a 306-example benchmark requiring joint visual perception and tabular reasoning. Fine-tuning Qwen2.5-VL-7B on TABLET achieves state-of-the-art on 11/15 table understanding benchmarks.

Problem

Existing VTU training datasets (e.g., MMTab) render table images from serialized text, producing synthetic visualizations that discard real-world visual features such as merged cells, background colors, font variations, and embedded images. Models trained on such synthetic renderings fail to generalize to naturally occurring tables, creating a train-test distribution mismatch. Furthermore, no large-scale, multi-task dataset preserving original table visualizations existed; prior datasets either covered a single task or lacked traceability to source examples.

Method

TABLET aggregates 4,066,851 instruction-tuned examples across 21 tasks from 14 seed datasets (TURL, ToTTo, TabFact, WikiTableQuestions, HybridQA, HiTab, PubTabNet, TabMWP, TAT-QA, InfoTabs, WikiBIO, FeTaQA, MMTab, DocStruct4M). Original table images are retrieved using Wikipedia's archiving API, with the correct table identified via Levenshtein edit-distance similarity (\(\geq 0.7\) threshold) against markdown-serialized tables from the seed datasets. Screenshots are rendered with Firefox in headless mode at 96 PPI. Tables unavailable via retrieval (11.5%) are synthetically rendered from HTML. Every example ships with: the original rendered image, an HTML serialization (renderable back to the original visual), source/table IDs for traceability, and the instruction text. For evaluation, the VisualTableQA benchmark was hand-annotated: human annotators selected high-visual-complexity tables using an automated visual complexity scorer, then wrote questions answerable only through joint visual and structural reasoning; questions solvable from lossy synthetic images alone were filtered out.

Key Contributions

  • TABLET dataset: 4M examples, 2M unique tables, 88% with original lossless visualizations, spanning 21 tasks across 7 task categories.
  • Unified extensible format: each example includes instruction, atomic data, HTML serialization, source ID, and metadata; HTML renders back to the original visualization enabling cell highlighting preservation.
  • VisualTableQA benchmark: 306 manually curated examples requiring joint visual perception and table understanding, unavailable in any prior resource.
  • SFT results: fine-tuning Qwen2.5-VL-7B on TABLET achieves state-of-the-art on 11/15 table understanding benchmarks, including held-out tasks.
  • Robustness finding: training on original (org) visualizations substantially reduces degradation when evaluating on real-world tables vs. synthetic ones (DegScore −6.63 for org-only vs. −28.90 for zero-shot).

Results

  • Robustness (Table 2, Qwen2.5-VL-7B): zero-shot DegScore = −28.90; TABLET-B\(_\text{synth}\) = −22.35; TABLET-B\(_\text{org}\) = −6.63; TABLET-B\(_\text{mix}\) = −7.87 — training on original visualizations cuts aggregate degradation by ~4.4× relative to zero-shot.
  • On individual benchmarks (org evaluation), TABLET-B\(_\text{mix}\) outperforms other variants on WikiTQ (56.5 exact match), HiTab (26.9 F1), TabFact (70.0 accuracy), FeTaQA (28.5 BLEU).
  • Fine-tuning on TABLET (full distribution) achieves state-of-the-art on 11 out of 15 table understanding benchmarks including held-out tasks.
  • VisualTableQA: fine-tuning on TABLET improves performance on this unseen benchmark relative to zero-shot and MMTab-trained baselines (specific numbers truncated in provided text).
  • TABLET is 9.38× larger than MMTab (4,066,945 vs. ~433,376 examples) and covers more of each seed dataset (e.g., 91.8% of ToTTo vs. MMTab's 12.4%).

Limitations

  • VisualTableQA is small (306 examples), limiting statistical power for fine-grained evaluation.
  • Wikipedia-sourced tables (61.5%) dominate; domain diversity outside Wikipedia and scientific PDFs is limited.
  • 11.5% of tables are still synthetic due to failed retrieval (low Levenshtein similarity), introducing the same train-test gap the dataset aims to eliminate for those instances.
  • Backbone exploration and advanced SFT strategies are explicitly out of scope; all main results use a single backbone (Qwen2.5-VL-7B).
  • The visual complexity scorer used to select VisualTableQA examples is automated and its calibration is not independently validated in the main text.
  • Table interpretation tasks (entity linking, column type annotation, relation extraction) dominate training at 54.2%, potentially skewing learned representations.

Relevance to Vision-Language Models

TABLET directly addresses a critical bottleneck for VLMs deployed in pixel-based agentic settings: the lack of large-scale training data grounded in real-world table visuals. The degradation analysis (DegScore) quantifies the gap between training on synthetic vs. original visualizations, providing a concrete diagnostic for VLM robustness on structured visual content. By enabling supervised fine-tuning of general-purpose VLMs (Qwen2.5-VL, Gemma 3) on 4M diverse examples, TABLET pushes toward a foundation model paradigm for table understanding rather than task-specific architectures. The VisualTableQA benchmark operationalizes a harder evaluation target — joint visual+structural reasoning — that will likely become a standard probe for measuring VLM progress on semi-structured document understanding.