All Explanations are Wrong, But Many Are Useful: Exploring the Rashomon Explanation Set with Large Language Models¶

🕒 Published (v1): 2026-07-10 15:16 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

RashomonLLM argues that the accuracy–explainability trade-off is an artifact of treating explanation and prediction as separate objectives. It proposes a three-agent LLM agentic workflow (Explanation–Prediction–Reflection, EPR) that iteratively aligns explanations with predictions, building a set of faithful natural-language explanations (the Rashomon Explanation set) rather than a single one. Across churn, survival, and large-scale CTR tasks, the coupled framework improves both explanation quality and predictive accuracy over state-of-the-art baselines.

Problem¶

Three compounding failures of existing XAI methods: (1) any single simplifying explanation fails under a suitably small distribution shift (Theorem 1); (2) scalar attributions (SHAP, LIME) cannot represent conditional or nonlinear relationships that require natural-language descriptions; (3) post-hoc explanations are decoupled from prediction, so they cannot be verified against true data patterns and do not improve model performance.

Method¶

Rashomon Explanation paradigm. Define the Rashomon Explanation set as \(\mathcal{R}_{\text{explanation}}(\mathcal{F}, \theta) := \{E : d(E, E_S) \leq \theta;\; f_E \in \mathcal{R}_{\text{set}}(\mathcal{F}, \theta)\}\) — explanations that are simultaneously faithful (within distance \(\theta\) of ground truth \(E_S\)) and useful (the explanation-guided model \(f_E\) achieves near-optimal risk).

RashomonLLM (EPR workflow). Three LLM agents operate in alternating rounds: - Explanation Agent generates natural-language feature–outcome explanations conditioned on current predictions. - Prediction Agent produces predictions conditioned on the current explanation. - Reflection Agent revises the prompting strategy (meta-level update) to reduce the explanation–prediction gap; implements double-loop learning (Argyris & Schön 1978).

The loop runs until prediction error stabilizes and explanations stop updating. Multiple independent runs of EPR explore different explanation trajectories; the union forms the Rashomon Explanation set, and the final explanation is their aggregate.

Theorem 4 (not fully reproduced in the excerpt) establishes convergence and recovery of the full Rashomon Explanation set under explicit conditions.

Fidelity bounds excess risk: \(L(f_{Q^*_E}) - L(f^*) \leq \sqrt{2B^2 \cdot \frac{1}{2}\varphi(d(E, E_S))}\) (Theorem 2), so improving explanation fidelity monotonically tightens the gap to the optimal predictor.

Key Contributions¶

Formal proof that no single explanation model is robust to all distribution shifts (Theorem 1), motivating ensemble explanations.
Rashomon Explanation definition: a set-valued analogue of the Rashomon Set applied to explanations, with non-emptiness theorem (Theorem 3) under mild assumptions.
EPR agentic workflow coordinating three LLM agents via alternating optimization with a meta-level Reflection Agent.
Theorem 2: explanation fidelity upper-bounds excess prediction risk, establishing theoretical link between XAI quality and predictive performance.
Empirical demonstration that coupling explanation with prediction improves, rather than degrades, accuracy.

Results¶

Customer churn (Kaggle): RashomonLLM significantly outperforms SHAP, LIME, ROLEX, and black-box baselines on both AUC/accuracy and explanation fidelity metrics (exact numbers not reproduced in the provided excerpt).
Clinical survival regression (Kaggle): Similar improvements over interpretable baselines (Logistic Regression, Decision Tree) and post-hoc methods.
CTR prediction (Kuaishou live-streaming, large-scale logs): Surpasses specialized deep CTR models.
Synthetic credit-score ablation (Table 4): GPT-4o with ground-truth explanation achieves RMSE 0.81 / MAE 0.63; no explanation: RMSE 5.97 / MAE 4.49; 10%-noise explanation: RMSE 3.36 / MAE 1.78 — demonstrating monotone fidelity-performance coupling.
Ablation and robustness analyses confirm gains persist across distribution shifts, temporal splits, and random seeds.
Calibration analysis attributes accuracy gains specifically to explanation fidelity, not model scale.

Limitations¶

Theorem 4 (convergence guarantee) requires explicit conditions that are not fully detailed in the provided excerpt; practical convergence may depend on LLM backbone quality.
"Usefulness" in Definition 3 is population-level (predictive utility), deliberately excluding human-perceived usefulness — behavioral validity of explanations is left to future work.
Relies on LLM backbone (e.g., GPT-4o); cost and latency of iterative multi-agent EPR rounds may be prohibitive at inference time for very high-throughput settings.
Ground-truth explanation \(E_S\) used in theoretical bounds is unobservable in practice; the fidelity distance \(d(E, E_S)\) can only be approximated.
Evaluated on tabular data; generalization to unstructured inputs (images, text) is not addressed.

Relevance to Harnesses / Meta-Harnesses¶

RashomonLLM is a concrete instance of a multi-agent iterative harness: three specialized LLM agents (Explanation, Prediction, Reflection) are coordinated by a meta-level Reflection Agent that rewrites the prompting strategy — a double-loop orchestration pattern directly analogous to meta-harness designs that separate object-level execution from strategy-level revision. The paper provides formal convergence guarantees for such a harness (Theorem 4), which is rare in the agentic-workflow literature and directly useful for researchers asking when iterative multi-agent loops provably terminate and recover target solution sets. The EPR alternating-optimization structure also exemplifies how harnesses can enforce alignment between subsystems (explanation ↔ prediction) that would otherwise optimize independently and diverge.