Explaining Decentralized Multi-Agent Reinforcement Learning Policies¶

🕒 Published (v1): 2025-11-13 15:30 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper introduces the first methods to summarize and explain decentralized Multi-Agent Reinforcement Learning (MARL) policies, where agents act on local observations without a global controller. The approach constructs Hasse diagram-based policy summarizations and generates natural-language query-based explanations ("When," "Why Not," "What") that explicitly handle nondeterminism and partial observability. User studies confirm that both artifacts significantly improve human understanding and question-answering accuracy over baselines.

Problem¶

Existing XRL explanation methods for MARL assume centralized execution or full joint observability (CTCE/CTDE from the explanation side), and thus cannot capture the uncertainty, nondeterminism, and inter-agent ordering ambiguity inherent in decentralized execution where each agent has only a local policy and local observations. No prior work generates policy summarizations or query-based explanations for decentralized MARL.

Method¶

Policy Summarization (HDS): Given per-agent trajectories from decentralized execution, Algorithm 1 incrementally constructs a Hasse diagram — a directed acyclic graph (DAG) where nodes represent (possibly jointly) completed tasks annotated with responsible agents, and edges encode partial-order precedence constraints. Transitive reduction yields a compact representation. Correctness (paths respect each agent's observed task order) and completeness (every agent's full trace corresponds to some path) are proven. Complexity is O(N·|T|² + |T|⁴).

Query-Based Explanations (HD-When/Why-Not/What): Abstract states are derived from Hasse diagrams across multiple simulated episodes. An uncertainty dictionary is built via partial comparability graphs: tasks with no ordering relation to the query node are marked uncertain. Boolean feature vectors (with conservative encoding of uncertain features) are fed into Quine-McCluskey minimization to extract minimal distinguishing formulas. These are translated into natural language with "must" (certain) and "may" (uncertain) conditions. "Why Not" queries invert target/non-target sets; "What" queries enumerate certain/uncertain successors directly from child nodes and partial comparability graphs.

Key Contributions¶

First method to generate policy summarizations for decentralized MARL, capturing inter-agent coordination and task ordering under stochastic, asynchronous execution.
Novel uncertainty dictionary mechanism using partial comparability graphs to represent unordered task dependencies in explanations.
Three query types ("When," "Why Not," "What") extended to decentralized settings with explicit uncertainty quantification via "may" conditions.
Algorithm-agnostic: works on both CTDE (SEAC) and DTDE (IA2C) trained policies.
Empirical validation across 4 MARL benchmark domains (SR, LBF, RW, PP) scaling to 9 agents and 19 tasks; all methods run in under 1 second.
Two IRB-approved user studies demonstrating objective and subjective improvements.

Results¶

Summarization compactness (CTDE, 100 episodes): - HDS: 7–20 nodes, 6–19 edges per diagram vs. baseline: 265–1,274 nodes, 258–1,270 edges (side-by-side per-agent graphs).

Summarization user study (N=20): - HDS: M=4.25/6 correct vs. baseline: M=3.1/6 (paired t-test: t(19)=4.2, p≤0.01, d=0.96). - HDS rated significantly higher in completeness (W=16.0, p≤0.04, r=-0.33); no significant difference on other 6 Likert dimensions.

Explanation user study (N=21), all comparisons p≤0.01: - "When": t(20)=9.65, d=2.16 - "Why Not": t(20)=13.23, d=2.96 - "What": t(20)=12.05, d=2.69 - All seven subjective quality metrics (understanding, satisfaction, detail, completeness, actionability, reliability, trust) rated significantly higher for HDE vs. baseline. - No significant increase in response time.

Limitations¶

Quine-McCluskey minimization has worst-case exponential complexity O(3^|Fq| / ln|Fq|) in the number of features, limiting scalability to large feature sets.
User studies are small (N=20/21), university-recruited, and survey-based — not real-world operator deployments.
DTDE (IA2C) experiments are restricted to smaller domain configurations (LBF(5,5), RW(3,4)) due to IA2C convergence failures in larger environments.
Summarizations reflect observed or simulated trajectories; behavioral coverage depends on the number and diversity of sampled episodes.
No support for continuous-action spaces or non-cooperative (competitive) MARL settings.
Natural language templates are structured/fixed, not flexible to free-form user queries.

Relevance to Agentic AI / LLM Agents¶

Interpretability of multi-agent systems is a direct prerequisite for trustworthy deployment of agentic AI — operators must understand what a team of autonomous agents is doing and why. This work advances the XAI-for-agents frontier by tackling the hard case of decentralized execution, where individual agents have no global view, mirroring realistic deployments of LLM-based agent pipelines with limited inter-agent communication. The uncertainty-aware explanation framework (certain vs. "may" conditions) is directly relevant to agentic systems that must convey confidence to human supervisors. The future direction of integrating LLMs for more expressive query handling explicitly bridges this work to the LLM agent paradigm.