Towards Corpus-Grounded Agentic LLMs for Multilingual Grammatical Analysis¶

🕒 Published (v1): 2025-11-28 21:27 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

UDagent is a fixed-plan agentic LLM system that answers multilingual grammatical questions by generating Python code to analyze Universal Dependencies (UD) treebanks and aggregating the results. Evaluated on 13 word-order tasks across up to 179 languages, it outperforms both a WALS majority baseline and a raw GPT-5 baseline on most tasks. The paper also releases a 2,275-pair multilingual benchmark for future grammatical analysis agents.

Problem¶

Corpus-based linguistic analysis requires complex scripting, specialized tools, and deep methodological expertise, making large-scale or exploratory grammatical inquiry labor-intensive and inaccessible. Existing LLM-corpus integrations (e.g., text-to-CQL, corpus chatbots) are task-specific and lack a systematic, evaluation-grounded framework for general grammatical reasoning over annotated corpora.

Method¶

UDagent implements a fixed four-step pipeline: 1. Task conversion — restate the general grammatical question (e.g., "What is SVO order in French?") as a sentence-level subtask ("Given a sentence, determine SVO order"). 2. Code generation — prompt GPT-5 (high reasoning effort) with the CoNLL-U format spec and valid WALS answer options to produce a Python function that accepts a single UD-annotated sentence and returns an order label or None. 3. Resource analysis — execute the generated function over every sentence in the target language's UD training split; collect per-sentence labels. 4. Result summarization — compute the frequency distribution of non-None labels; dominant order = argmax; full distribution = detailed answer.

Evaluation uses three complementary metrics: classification accuracy (dominant order), coverage F1 (set of attested orders, averaged over languages), and Hellinger distance \(H(P,C) = \frac{1}{\sqrt{2}}\sqrt{\sum_i(\sqrt{P_i}-\sqrt{C_i})^2}\) (distributional fidelity). Ground truth is produced by hand-written Python scripts on UD v2.16 test splits.

Key Contributions¶

A structured evaluation framework covering 13 WALS-aligned word-order tasks across up to 179 languages with three-layered gold annotations (dominant order, attested orders, full distribution).
A publicly released benchmark of 2,275 feature–language pairs derived from UD v2.16.
The UDagent prototype demonstrating that a simple fixed-plan code-generation agent already surpasses both a typological prior (WALS majority) and a parametric-only LLM baseline on most tasks.
Cross-linguistic analysis showing perfect dominant-order accuracy for 112/159 languages and diagnosing failure modes (sparse treebanks, near-tie distributions, task-interpretation mismatches, code generation failures).

Results¶

UDagent achieves best accuracy on 9/13 tasks, best coverage F1 on 10/13, best Hellinger distance on 9/13.
When successful, accuracy and F1 typically exceed 0.9; Hellinger distance often below 0.1.
Fails completely on features 144A and 144B (accuracy 0.056 / 0.000, Hellinger 0.971 / 1.000) due to code-generation producing syntactically valid but semantically incorrect scripts.
GPT-5 baseline (no corpus grounding) achieves best score on 4/13 accuracy tasks, showing non-trivial parametric linguistic knowledge but leaving clear room for grounded improvement.
Majority (WALS) baseline accuracy ranges from 0.069 (feature 89A) to 0.905 (82A), confirming high task-level imbalance variation.

Limitations¶

Fixed plan lacks adaptability; complex or ambiguous tasks that need dynamic replanning are not handled.
Code generation silently fails for features requiring complex syntactic reasoning (144A/B), with no fallback or self-repair loop.
Evaluation restricted to word-order phenomena; generalization to other grammatical features is asserted but not demonstrated.
Relies on GPT-5 (closed, costly); open-source model substitution is planned but not evaluated.
Ground-truth scripts are language-agnostic by design, which can diverge from linguistically informed interpretations (e.g., Dutch pronoun vs. noun subject ambiguity).
Small or domain-narrow UD treebanks (Buryat, Komi Zyrian) cause data-sparsity failures independent of agent quality.

Relevance to Harnesses / Meta-Harnesses¶

UDagent is a concrete instantiation of the fixed-plan code-generation harness pattern: a static pipeline that wraps an LLM's code synthesis capability with corpus execution and result aggregation, abstracting manual scripting from the end user. The evaluation framework itself functions as a meta-harness — it defines task schemas, ground-truth generation scripts, and multi-metric scoring that can be reused to benchmark any future grammatical analysis agent, making it a reusable scaffolding layer rather than a one-off tool. The paper's explicit separation of the harness architecture (plan, code gen, execution, summarization) from the underlying model is directly relevant to harness design principles: model-swappability and deterministic pipeline structure. The failure analysis (silent code-gen errors producing valid-but-wrong functions) highlights a key harness robustness gap — the absence of a verification or self-repair loop — that meta-harness designers need to address.