Skip to content

Can LLMs Write Faithfully? An Agent-Based Evaluation of LLM-generated Islamic Content

🕒 Published (v1): 2025-10-28 14:05 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A dual-agent framework (quantitative + qualitative) evaluates three LLMs—GPT-4o, Ansari AI, and Fanar—on 50 Islamic scholarly prompts, scoring outputs across citation accuracy, theological fidelity, and stylistic quality. GPT-4o leads quantitatively (3.90/5 mean) while Ansari AI leads qualitatively (116/200 "Best" verdicts); all models remain unreliable for faithful citation and reference attribution.

Problem

Existing evaluation metrics (BLEU, ROUGE) measure surface overlap but cannot assess theological accuracy, Qur'anic/Hadith citation integrity, or culturally appropriate tone. No prior pipeline unified theological verification with stylistic evaluation for Islamic content, despite the domain carrying high stakes for misinformation and spiritual harm—analogous to failures documented in legal and medical LLM evaluation.

Method

The framework consists of two coordinated agents:

  1. Quantitative Agent: Uses OpenAI o3 augmented with three tools (Qur'an Ayah retrieval, Internet Search, Internet Extract) to score each essay 1–5 across six criteria: Structural Coherence, Thematic Focus, Clarity, Originality, Islamic Accuracy, and Citation/Islamic Source Use. References are verified and flagged (confirmed / partially confirmed / unverified / refuted), with score deductions for non-confirmed citations compiled in an accuracy_verification_log.

  2. Qualitative Agent: Processes all three model responses simultaneously using XML-tagged segmentation (<R1>, <R2>, <R3>) and evaluates side-by-side across five dimensions (Clarity & Structure, Islamic Accuracy, Tone & Appropriateness, Depth & Originality, Comparative Reflection), issuing "Best"/"Worst" verdicts backed by textual evidence.

Fifty prompts spanning five domains (Fiqh, Tafsir, Ulum al-Hadith, Aqidah, Adab) were collected from authentic Islamic scholar blogs and sent verbatim to each model.

Key Contributions

  • Dual-agent evaluation harness combining tool-augmented citation verification (quantitative) with pairwise stylistic analysis (qualitative)
  • Six-dimension quantitative scoring rubric extended from general essay evaluation to cover Islamic theological fidelity
  • First systematic comparison of GPT-4o, Ansari AI, and Fanar on faith-sensitive long-form generation
  • Modular, interpretable blueprint explicitly designed for transfer to other high-stakes domains (medicine, law, journalism)
  • 150-essay dataset (50 prompts Ă— 3 models) with full prompt-response pairs archived publicly

Results

  • GPT-4o highest quantitative mean: 3.90/5 (std = 0.589); Ansari AI: 3.79/5; Fanar: 3.04/5 (std = 0.923)
  • Islamic Accuracy: GPT-4o 3.93, Ansari AI 3.68, Fanar 2.76
  • Citation: GPT-4o 3.38, Ansari AI 3.32, Fanar 1.82
  • Theme score: GPT-4o 4.43 (highest single dimension); Fanar Originality 2.73 (lowest)
  • Qualitative "Best" verdicts: Ansari AI 116/200, GPT-4o 84/200, Fanar 0/200
  • Fanar "Worst" verdicts: 193/200 across qualitative dimensions
  • GPT-4o strongest qualitative dimension: Tone & Appropriateness (48 Best)
  • Ansari AI strongest: Clarity & Structure (41 Best) and Islamic Accuracy (42 Best)

Limitations

  • Pilot scale: only 50 prompts, single human evaluator sanity-check (no multi-scholar validation panel)
  • Evaluator LLM homogeneity: o3 used as sole quantitative judge; no cross-family inter-evaluator agreement measured
  • Arabic-primary models (Fanar) evaluated predominantly in English, disadvantaging their design intent
  • No stratified coverage across madhahib or classical vs. contemporary jurisprudence
  • Classical Islamic corpora (Shamela, OpenITI) not systematically integrated; extent of inclusion in model pretraining unknown
  • Fanar's architectural constraints (9B parameters, 4,096-token context) confound model quality with resource constraints

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instance of an evaluation meta-harness: two specialized agents (citation verifier + qualitative comparator) are orchestrated with shared tooling to produce complementary, convergent assessments of downstream LLM outputs. The modular design—tool-augmented verification feeding structured logs, qualitative side-by-side with XML segmentation, composite scoring across independent dimensions—is a deployable harness pattern applicable beyond Islamic content to any high-stakes domain requiring multi-dimensional faithfulness auditing. The explicit claim of a "blueprint adaptable to other high-stakes domains" positions this as a generalizable harness architecture rather than a one-off evaluation, making it directly relevant to researchers building multi-agent quality-assurance pipelines.