Skip to content

NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NeedleInATable (NIAT) introduces the first long-context benchmark targeting structured tables, where each table cell is a "needle" to be retrieved via cell-locating or cell-lookup queries. Comprehensive evaluation of LLMs and MLLMs reveals a large performance gap between high-level downstream tabular tasks and this simpler cell-level perception task, exposing that existing models rely on shortcuts rather than genuine structural understanding. A strong2weak data synthesis method using 12K GPT-4o-generated CoT examples yields +14.55% average accuracy on four downstream benchmarks when fine-tuning 7–8B models.

Problem

Existing long-context benchmarks (e.g., Needle-in-a-Haystack) evaluate only unstructured text, while tabular benchmarks test high-level reasoning on partial table content. Neither assesses fine-grained, cell-level perception across entire long tables — the foundational capability required for reliable LLM-based table applications. Models that score well on complex tabular QA may be exploiting dataset shortcuts rather than genuinely understanding table structure.

Method

Benchmark construction: 750 real tables (flat, hierarchical, horizontal) sourced from WTQ, HiTab, and AIT-QA are serialized into Markdown, HTML, and image formats. 60% of cells per table are sampled to generate two query types: (1) cell-locating — retrieve cell at explicit (row, column) coordinates; (2) cell-lookup — retrieve cell via natural-language lookup (no aggregation). Cell-locating queries use template filling; cell-lookup queries are generated by GPT-4o with in-context learning and filtered via self-consistency. Total: 145K cell-locating + 142K cell-lookup = 287K queries; prompt lengths up to 120K tokens.

Data synthesis (strong2weak): GPT-4o generates NIAT queries plus detailed chain-of-thought reasoning traces on training-split tables. Six harder cell-lookup sub-tasks (e.g., cell-retrieval over the whole table) diversify the training signal. 6K examples per task type (12K total) are used to fine-tune Llama3.1-8B-Instruct and Qwen2.5-7B-Instruct.

Key Contributions

  • First long-context benchmark specifically targeting structured tables (NIAT), with 750 tables, 287K test cases, three table structures, three formats, and variable sizes up to 32×32.
  • Large-scale evaluation of generalist LLMs, tabular LLMs (StructLLM, TableGPT2), reasoning LLMs (DeepSeek-R1, QwQ-32B), GPT-4o, and 18 MLLMs, revealing a systematic "lost-in-the-middle-table" phenomenon and format/structure brittleness.
  • Discovery that LLMs employ two attention patterns (Multi-Slash, Local-Triangle) enabling semantic lookup but not structural locating.
  • Strong2weak CoT synthesis method: 12K training samples outperform tabular-specialist baselines trained on orders-of-magnitude more data on four downstream benchmarks.

Results

  • Cell-locating vs. cell-lookup gap: Best open-source LLM (Qwen3-30B-A3B) achieves 16.49% on cell-locating vs. 78.62% on cell-lookup; GPT-4o reaches 26.00% vs. 68.30%.
  • DeepSeek-R1 achieves SOTA at 65.91% cell-locating / 80.99% cell-lookup (overall 73.45%), far exceeding GPT-4o (47.15%) and QwQ-32B (42.58%).
  • MLLMs: Skywork-R1V3-38B leads at 24.50% cell-locating / 63.92% cell-lookup; GLM-4.1V-Thinking-9B reaches 25.85% / 75.12%.
  • Fine-tuning with 12K NIAT data: Llama3.1-8B +14.55% average accuracy on WTQ/TabFact/HiTab/TABMWP; Qwen2.5-7B +3.78%; both surpass TableGPT2 (trained on 2.36M samples + 86B CPT tokens).
  • Ablation: Combining both cell-locating + cell-lookup tasks yields best downstream gains; fine-tuning on 4 downstream datasets with CoT is comparable but not consistently better.
  • Performance degrades monotonically with table size (8×8 → 32×32), confirming a unique long-table challenge beyond standard NIAH.

Limitations

  • Cell-locating performance remains low even for the best models, and the "lost-in-the-middle-table" phenomenon is only partially mitigated by fine-tuning.
  • Evaluation of hierarchical tables is worse than flat/horizontal, and the benchmark's coverage of extremely deep hierarchies is limited by source datasets.
  • The strong2weak synthesis relies on GPT-4o quality and self-consistency filtering; errors in generated CoT traces may propagate.
  • NIAT queries are limited to single-cell retrieval with no aggregation; it does not directly evaluate multi-cell or multi-hop operations over long tables.
  • Image-format evaluation uses a smaller 120-table subset per structure, limiting statistical power for MLLMs.

Relevance to Vision-Language Models

NIAT directly stresses MLLMs' ability to process table images, exposing that fixed-size ViT patch tokenization disrupts structural integrity and causes systematic failures on cell-locating even when semantic lookup succeeds. The benchmark provides a controlled, fine-grained diagnostic for VLMs processing document and table images — a regime of growing practical importance. The finding that reasoning-augmented VLMs (Skywork-R1V3, GLM-4.1V-Thinking) substantially outperform standard instruction-tuned ones on this task informs the broader VLM design debate around test-time compute vs. architectural improvements. The image-format evaluation pipeline and MLLM analysis complement text-modality findings, making NIAT a useful testbed for future VLM table understanding work.