Skip to content

LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models

🕒 Published (v1): 2025-11-14 14:13 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LaoBench is the first large-scale, multidimensional benchmark (17,000+ instances) for evaluating LLMs in Lao, a low-resource Southeast Asian language. It covers culturally grounded knowledge, K12 education, and trilingual translation, with a contamination-resistant held-out subset for black-box evaluation. State-of-the-art LLMs including GPT-5-High and Gemini-2.5-Pro still lag substantially behind human experts, particularly on culturally grounded reasoning and translation fidelity.

Problem

Lao—despite being spoken by millions—is nearly absent from large-scale LLM evaluation suites. Existing SEA benchmarks rely heavily on translated English content, miss Lao-specific cultural grounding, and lack held-out evaluation to prevent leaderboard contamination. Lao's scriptio continua writing system (no word boundaries), tonal phonology, and Pali–Sanskrit loanwords create unique NLP challenges not addressed by generic multilingual benchmarks.

Method

LaoBench is constructed via a three-stage hybrid pipeline: 1. Raw data acquisition: K12 textbooks, government/legal documents, encyclopedic resources, and culturally grounded Lao articles. 2. Dataset construction: Expert linguists write MCQs (Lao-7k/Lao-10k) with calibrated distractors; Lao-500 open-ended prompts are selected from a large candidate pool using an LLM scorer (BenchBuilder-inspired) on dimensions of specificity, clarity, and domain depth, followed by topic-based clustering and balanced sampling. 3. Validation: Multi-stage pipeline combining human expert review (factual correctness, cultural appropriateness, distractor plausibility) with agent-assisted verification (duplicate/near-duplicate detection, semantic consistency checks, context-independence filtering, sensitivity screening).

Evaluation: MCQs are scored zero-shot with output normalization to A/B/C/D. Translation uses SacreBLEU with LaoNLP-aware tokenization. Lao-500 uses Arena-style pairwise comparison against GPT-5-High as baseline, with two independent judges (Gemini-2.5-Pro, Qwen3-Max) and bootstrap 95% CIs to control for judge bias.

Key Contributions

  • First multidimensional Lao benchmark with 17,000+ expert-curated instances spanning knowledge application, K12 education, and Lao–Chinese–English translation.
  • Contamination-resistant design: 10,000+ held-out MCQs (Lao-10k) with a black-box evaluation service returning only aggregated scores.
  • Scalable hybrid construction pipeline combining expert authoring with LLM-agent-assisted quality verification.
  • Comprehensive zero-shot benchmarking of SOTA open-source and closed-source LLMs, quantifying persistent gaps to human performance.

Results

  • K12 Education (accuracy): GPT-5-High ~90%, Gemini-2.5-Pro ~89%, Claude-Sonnet-4.5-thinking ~89%; open-source best: Qwen3-235B-A22B-Instruct-2507 ~86%. Human experts: ~98%.
  • Knowledge Application (accuracy): Closed-source models reach ~77–80%; open-source best ~71–75%. Human experts: ~98–99%.
  • Translation (BLEU): Best closed-source Gemini-2.5-Pro reaches mid-30s BLEU on easier subdomains; Culture & History and Social & Law subdomains are hardest for all models.
  • CoT prompting: Consistent gains on Knowledge Application and multi-step reasoning subdomains; minimal gain on rote K12 recall.
  • Lao-500 Arena: Model rankings differ from MCQ rankings; GPT-5-High used as baseline—Gemini-2.5-Pro achieves ~52% win rate; confidence intervals for closely ranked models overlap, indicating non-significant differences.
  • Judge bias: Qwen3-Max judge systematically inflates win rates for Qwen-family models vs. Gemini-2.5-Pro judge.

Limitations

  • Majority MCQ format may over-reward test-taking strategies and under-represent genuine open-ended reasoning.
  • BLEU is sensitive to Lao tokenization and penalizes valid paraphrastic translations; chrF++ reported as secondary metric only.
  • Arena-style evaluation anchors to a fixed baseline (GPT-5-High) and is subject to judge self-preference and family-correlated bias despite multi-judge mitigation.
  • Lao-10k black-box service not yet deployed at publication time.
  • Benchmark scope is restricted to Lao; generalization to other scriptio continua or tonal languages requires separate construction effort.

Relevance to Agentic AI / LLM Agents

LaoBench is directly relevant to agentic AI practitioners deploying LLM-based agents in multilingual or Southeast Asian contexts, as it quantifies the Lao-language capability ceiling of models like GPT-5, Gemini-2.5-Pro, and Claude—all commonly used as agent backbones. The hybrid construction pipeline itself exemplifies LLM agents used for automated quality assurance (duplicate detection, semantic consistency, sensitivity screening), offering a reusable pattern for benchmark-builders. The persistent human–model gap on culturally grounded reasoning highlights a reliability risk for agentic systems operating in Lao-language environments, where factual and cultural errors could propagate silently. Finally, the Arena-style evaluation protocol for open-ended generation on Lao-500 provides a methodological template for assessing agent-quality long-form responses in low-resource languages.