Skip to content

Know your Trajectory -- Trustworthy Reinforcement Learning deployment through Importance-Based Trajectory Analysis

🕒 Published (v1): 2025-12-07 16:52 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a framework for trajectory-level explainability in RL by ranking entire trajectories using a novel state-importance metric that combines Q-value advantage with a goal-proximity "radical term." Counterfactuals generated from critical states in top-ranked trajectories provide contrastive "Why this path and not that?" explanations. The V-Goal variant of the metric reliably identifies optimal trajectories where all counterfactual deviations lead to worse outcomes.

Problem

Existing Explainable RL (XRL) methods focus on local, single-step decisions, failing to explain an agent's long-term strategy. Prior trajectory-level approaches (e.g., HIGHLIGHTS, clustering-based methods) either summarize via discrete states or rely on cluster interpretability, neither providing holistic, contrastive explanations over full sequential paths from heterogeneous experience buffers that mix optimal and suboptimal trajectories.

Method

The framework defines a modified state-action importance metric I(s,a) = ΔQ(s) × R(s,a), where ΔQ(s) = max_a Q(s,a) − min_a Q(s,a) captures decision criticality and R(s,a) is a "radical term" encoding goal affinity. The best-performing radical term is V-Goal: r(s) = |V(s)/V(s_final)|, which normalizes the state-value function relative to the goal state's value. Trajectory importance is the mean I(s,a) over all state-action pairs. The pipeline (1) collects trajectories from a trained agent, (2) computes per-state importance, (3) ranks trajectories by aggregate score and selects the best-outcome trajectory from the top-k, then (4) generates counterfactuals by forbidding the original action at each state and letting the policy roll out, enabling contrastive comparison.

Key Contributions

  • Novel state-action importance metric combining Q-value advantage with a goal-affinity radical term, with four formulations evaluated (naive normalization, Bellman error, entropy-based confidence, value-based goal proximity).
  • Complete pipeline for ranking heterogeneous trajectory datasets to surface optimal agent behavior.
  • Counterfactual rollout mechanism providing contrastive trajectory-level explanations ("Why this path, not that?").
  • Empirical validation showing V-Goal consistently selects trajectories where all counterfactuals are demonstrably worse.

Results

  • Acrobot-v1: V-Goal top-5 avg. length 68.8 / avg. reward −67.8 vs. classic ΔQ: 70.0 / −69.0; V-Goal counterfactuals are all longer (worse) than original; classic ΔQ produces counterfactuals shorter than original.
  • LunarLander-v2: V-Goal top-5 avg. reward 207.13 / avg. length 319.2 vs. classic ΔQ: 116.87 / 1000.0 (time-limit failures); V-Goal is the only metric that consistently identifies successful landing trajectories (reward > 200); all other methods select failed/meandering attempts.
  • V-Goal counterfactuals in LunarLander uniformly yield lower rewards and probabilistically longer trajectories; classic ΔQ counterfactuals include paths with higher reward than the "original," indicating incorrect trajectory selection.

Limitations

  • Tested only on two discrete/discretized OpenAI Gym environments (Acrobot, LunarLander) with PPO; scalability to continuous high-dimensional state spaces or model-free policies without a Q-table is unvalidated.
  • Discretization required for continuous state spaces may lose fidelity.
  • Framework degrades when analyzing a fully trained optimal agent whose trajectories are near-uniform in quality—limited discriminative signal.
  • Assumes access to the agent's Q-function and value function; applicability when these are unknown (black-box agents) is left to future work (IRL-based extension proposed but not implemented).
  • KL-divergence radical term explored but discarded due to high sensitivity to reference distribution choice, indicating the metric search is not exhaustive.

Relevance to Agentic AI / LLM Agents

Trustworthy deployment of autonomous agents requires not just performance but interpretable, auditable behavior—this work directly addresses trajectory-level explainability, a gap relevant to any long-horizon agent (RL or otherwise). The counterfactual "Why this path and not that?" framing maps cleanly onto the kind of post-hoc justification needed for agentic LLM systems operating over multi-step tool-use or planning trajectories. The importance-ranking pipeline also offers a principled way to curate high-quality demonstration data from mixed experience buffers, relevant to RLHF and agent fine-tuning workflows. For researchers building or auditing RL-based agents in safety-critical domains, this provides a concrete mechanism for identifying and communicating the agent's decision logic at the policy level.