LLM Agents Implement an NLG System from Scratch: Building Interpretable Rule-Based RDF-to-Text Generators¶

🕒 Published (v1): 2025-12-20 13:16 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents a neurosymbolic framework where five LLM agents collaboratively write a rule-based RDF-to-text NLG system in pure Python, eliminating the need for supervised training data. The resulting system is fully interpretable, runs on a single CPU, and produces zero hallucinations in human evaluation while remaining competitive with neural baselines on reference-based metrics.

Problem¶

Neural RDF-to-text systems suffer from hallucinations, lack interpretability, require human-labeled reference texts for training, and demand substantial compute (GPU) at inference. Existing rule-based LLM approaches (e.g., template-based methods) either require reference texts or produce unwieldy numbers of templates (113,000+), limiting maintainability.

Method¶

Five specialized LLM agents collaborate in a test-driven loop to produce a single Python file implementing an NLGSystem class: - Test Engineer (TE): Generates ~50 input/output unit test pairs per predicate by prompting an LLM from in-domain RDF triples alone (no human references). - Software Architect (SA): Designs the high-level code structure (function signatures, responsibilities) given the predicate list. - Software Engineer (SE): Implements each function one-by-one, given the design and failed tests as feedback. - Evaluator: Executes the program against unit tests and uses an LLM (always Llama 3.3 70B) to binary-judge output correctness; early-stops at 5 failures. - Code Analyst (CA): Analyzes failures and decides whether to trigger full redesign (back to SA) or partial refactoring (back to SE for specific functions).

Training terminates when all unit tests pass or the iteration limit (25 for open-source, 10 for GPT) is reached. The final artifact is a ~168-line standalone Python program requiring no LLM at inference. Tested with GPT-4.1, Qwen 3 235B, Qwen 2.5 72B, and Llama 3.3 70B (the latter three at 4-bit via Ollama).

Key Contributions¶

First application of multi-agent LLM code synthesis to NLG system construction, producing interpretable, reference-free RDF-to-text generators.
Unsupervised training pipeline: only in-domain RDF triples are required; no human-written references.
35× CPU speedup over BART on GPU; 272× over BART on CPU (7s vs. 1910s for WebNLG test set).
Zero major or minor hallucinations detected in human evaluation (100 instances, Cohen's κ = 0.83), vs. 40% major hallucination rate for fine-tuned BART.
Formal interpretability evaluation: two software engineers traced any output word to its source code line in under 10 seconds on average, and fixed omission bugs in under 5 minutes each.
Ablation confirming both the dynamic SA-generated design and the auto-generated unit tests (vs. real training data) are critical to performance.

Results¶

WebNLG test set (reference-based, Table 1): - GPT-4.1-trained system: METEOR 0.7069, BLEURT 0.1841 — beats fine-tuned BART (0.6791 METEOR, 0.1275 BLEURT) and prompted Llama 3.3 70B on both. - BLEU (0.3934) and BERTScore (0.9291) slightly below fine-tuned BART (0.4352, 0.9308) overall, but our system beats BART on OOD BLEU (0.3615 vs. 0.3052). - Qwen 3 235B closely tracks GPT-4.1; smaller models degrade noticeably.

OpenDialKG (Table 2): - Fine-tuned BART dominates reference-based metrics (BLEU 0.9372 vs. our 0.3144–0.3472), reflecting the importance of in-domain sentence structure; all our models outperform prompted Llama 3.3 70B.

Reference-less / hallucination (Tables 4–5): - GPT-4.1 system: additions (hallucinations) 0.029 on WebNLG vs. 0.510 for BART; 0.013 on OpenDialKG vs. 0.052 for BART. - Human eval: 0% minor and major hallucinations for our system vs. 22%/40% for BART. - Grammaticality slightly lower than prompted Llama 3.3 but comparable to or better than BART.

Inference time (Table 3): 7s CPU vs. 249s BART on GPU, 1910s BART on CPU.

Limitations¶

System may still produce omissions; omission rate (0.06 on WebNLG human eval) is comparable to prompted LLM but nonzero.
Programs are not robust to schema drift: GEM datasets with different date formats caused runtime errors and reduced performance.
Smaller open-source LLMs (Llama 3.3 70B, Qwen 2.5 72B) as training agents produce substantially weaker systems; powerful LLMs appear necessary.
Inference-time generalization is limited to predicates seen during training; entirely new predicates require retraining.
Small-scale human evaluation (100 instances, 6 annotators) limits statistical power.

Relevance to Agentic AI / LLM Agents¶

This work is a clean demonstration of a multi-agent code synthesis pipeline where agents with complementary roles (architect, engineer, tester, evaluator, analyst) collaborate in an iterative TDD loop to produce a deployable artifact — a paradigm directly applicable to any software generation task. It operationalizes test-driven development as the convergence criterion for agentic code writing, showing that LLM-generated unit tests (without ground-truth references) can guide iterative refinement effectively. The neurosymbolic framing — use agents to build a system that then operates without LLMs — is a compelling pattern for deploying interpretable, cheap-inference AI components. The CA agent's redesign-vs-refactor decision mirrors meta-level planning in hierarchical agent systems, making this a relevant architectural reference for agent-orchestration research.