Skip to content

SPAN: Benchmarking and Improving Cross-Calendar Temporal Reasoning of Large Language Models

🕒 Published (v1): 2025-11-13 05:57 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SPAN is a benchmark for evaluating LLM cross-calendar temporal reasoning across six calendar systems (Gregorian, Chinese Lunar, Islamic, Hebrew, Shaka, Persian), spanning dates from 1960–2060. Evaluated SOTA LLMs average only 34.5% accuracy, but a tool-augmented Time Agent reaches 95.31% by combining GPT-4o with a code interpreter and a calendar conversion API.

Problem

All prior temporal reasoning benchmarks are Gregorian-centric and time-invariant (fixed reference date), preventing evaluation of (1) non-Gregorian calendar reasoning critical for multicultural applications, and (2) temporal robustness across past/future dates. Data contamination is also a concern since fixed benchmarks can leak into training corpora.

Method

SPAN uses a template-driven dynamic instance generation protocol: given a user-specified Gregorian date, it (1) converts it into five non-Gregorian equivalents via a unified search_calendar interface, (2) matches these against a library of question–code template pairs spanning date-based and festival-based tasks in both polar and content question formats, (3) instantiates templates by filling offset variables (nd, nw, ny), and (4) executes Python code snippets to derive ground-truth answers automatically. This yields 1,780 instances per evaluation date across 21 dates (every 5 years, 1960–2060), totaling 37,380 instances. The Time Agent improvement uses GPT-4o with few-shot prompting to generate executable Python code invoking search_calendar, runs it via a code interpreter, then feeds the result back to GPT-4o for final answer generation.

Key Contributions

  • SPAN benchmark: 10 cross-calendar reasoning directions, 2 reasoning types (date-based, festival-based), 2 question formats (polar, content), across 6 calendars
  • Template-driven dynamic evaluation protocol enabling time-variant, contamination-resistant assessment at arbitrary Gregorian dates
  • Identification of two failure modes: Future-Date Degradation (accuracy drops sharply for post-training-cutoff dates) and Calendar Asymmetry Bias (Gregorian-to-Others consistently outperforms Others-to-Gregorian by 4–17%)
  • LLM-powered Time Agent achieving 95.31% average accuracy via tool-augmented code generation

Results

  • Average LLM accuracy across all models and evaluation dates: 34.5%; no model exceeds 80%
  • Per-model averages: GPT-4o 43.01%, Claude-3.7-Sonnet 48.40%, DeepSeek-V3 50.64%, Qwen-2.5-72B 28.70%, Llama-3.3-70B 24.39%, Gemini-1.5-Pro 11.94%
  • Time Agent: 95.31% average accuracy (vs. OpenAI-o1 at 59.29%, GPT-4o w/ RAG at 43.69%, GPT-4o alone at 43.01%)
  • Polar questions outperform content questions by 3.13–37.08% (18.86% average), expected given 50% random baseline
  • Festival-based reasoning outperforms date-based by 2.87–12.60% for most models, attributed to higher festival prevalence in pretraining data
  • GPT-4o evaluator agreement with human annotations: Spearman/Kendall/Cohen's Îş all 0.95–1.00

Limitations

  • Time Agent relies on GPT-4o as the backbone; errors stem from code execution failures and logical bugs in generated code
  • Polar questions are trivially biased: all gold answers are "Yes" (dect is always set to the correct date), inflating polar accuracy and not testing negative cases
  • Calendar coverage limited to six systems; many global calendars (Ethiopian, Coptic, Tibetan, etc.) excluded
  • Future-Date Degradation is diagnosed but not addressed intrinsically—Time Agent sidesteps it via external tools rather than improving model knowledge
  • RAG via Bing Search yields only +0.68% on future dates, and the method is not explored more deeply

Relevance to Agentic AI / LLM Agents

The Time Agent design is a clean instantiation of the tool-use + code-generation agent paradigm: LLM generates executable code, a code interpreter executes it, and the result is fed back into the LLM for final reasoning—achieving a 52-percentage-point lift over pure LLM reasoning (95.31% vs. 43.01%). This directly supports the broader finding that LLMs should offload precise symbolic computation to tools rather than attempt it internally, a central design principle for capable LLM agents. The benchmark also highlights knowledge cutoff sensitivity (Future-Date Degradation), a practical failure mode relevant to any agent reasoning about real-world temporal context. The search_calendar tool-use pattern exemplifies how narrow, well-specified external APIs can dramatically expand agent reliability in domain-specific tasks.