Skip to content

Abstract Counterfactuals for Language Model Agents

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Token-level counterfactual inference (TLCF) for LM agents fails because token identities shift meaning across contexts (e.g., "choice 2" means different things in different game states). This paper introduces Abstract Counterfactuals (ACF), a framework that performs counterfactual inference over a semantic abstraction variable Y rather than raw tokens, requiring only black-box LLM access. ACF achieves significantly better semantic consistency than TLCF on choice-based games and text-generation tasks.

Problem

Existing counterfactual inference methods for LMs operate at the token level using Gumbel-Max SCMs, which always increase the counterfactual probability of the observed token regardless of context. In LM agents, action tokens are context-dependent: the same token ("2", "choice 1") can encode opposite high-level behaviors in different states. TLCF thus carries no reliable semantic information into the counterfactual setting, producing biased or meaningless results in both choice-based and open-text environments.

Method

ACF augments the LM agent SCM with an abstraction variable Y_t = f_Y(A_t, S_t, U_t^Y) that captures the high-level meaning of action A_t in state S_t. Counterfactual inference proceeds in three steps: 1. Abduction: infer the posterior over the exogenous noise U^Y conditioned on the observed abstraction value y (not on the raw token a). 2. Counterfactual Y′: plug the posterior U^Y into the counterfactual state s′ to obtain a distribution over Y′. 3. Mapping back: derive the counterfactual action A′ by reweighting the LLM's action distribution P(A|s′) by P(Y′|s′, a′), marginalizing over Y′.

This bypasses abduction on the token-level mechanism f_A entirely, treating the LLM as a black box. Abstractions Y can be supervised (expert-defined classifiers, e.g., profession labels, ethical behavior annotations) or unsupervised (auxiliary LLM performs concept discovery). The paper proves ACF satisfies interventional consistency: averaging ACF counterfactuals over the prior on U^Y recovers the interventional distribution.

Key Contributions

  • Introduction of the Abstract Counterfactuals (ACF) framework: semantic abstraction variable Y interposed between token-level actions and counterfactual inference in an LM agent SCM.
  • Formal three-step inference procedure (abduction → counterfactual Y′ → back-mapping) that requires only black-box LLM sampling.
  • Support for both supervised (classifier-based) and unsupervised (LLM-driven concept discovery) abstraction construction.
  • Proof of interventional consistency for ACF.
  • Evaluation on three benchmarks: MACHIAVELLI (choice-based ethical behavior), Bios (gender-steering, latent-space intervention), GoEmotions (emotion tracking, token-level prompt intervention).
  • Three evaluation metrics: Abstraction Change Rate (ACR), Counterfactual Probability Increase Rate (CPIR), Semantic Tightness (ST).

Results

Bios dataset (latent-space gender-steering, GPT2-XL, n=250): - ACF supervised: ACR=0.04, CPIR=0.98 vs. TLCF: ACR=0.40, CPIR=0.59 - ACF unsupervised: ACR=0.12, CPIR=0.98 vs. TLCF: ACR=0.38, CPIR=0.73 - ST win rate (ACF over TLCF): 0.78 (p<0.001, t=13.04 supervised, t=10.95 unsupervised)

GoEmotions dataset (token-level prompt intervention): - GPT2-XL supervised: ACR=0.02, CPIR=0.96 vs. TLCF: ACR=0.32, CPIR=0.68; ST win rate=0.76 - GPT2-XL unsupervised: ACR=0.05, CPIR=0.97 vs. TLCF: ACR=0.37, CPIR=0.67; ST win rate=0.82 - Llama-3.2-1B supervised: ACR=0.27, CPIR=0.87 vs. TLCF: ACR=0.54, CPIR=0.48; ST win rate=0.76 - Llama-3.2-1B unsupervised: ACR=0.41, CPIR=0.75 vs. TLCF: ACR=0.67, CPIR=0.47; ST win rate=0.81

MACHIAVELLI: Qualitative case studies show ACF correctly redistributes counterfactual probability mass to actions consistent with observed ethical annotations; TLCF fails when action-space cardinality varies across factual/counterfactual states.

Limitations

  • Computational cost: requires multiple black-box LLM samples to estimate distributions; scales poorly with action-space size.
  • Abstraction design burden: defining a meaningful Y is non-trivial and domain-dependent; poorly calibrated abstractions (too coarse or too fine) degrade results.
  • Unsupervised concept discovery quality depends on the auxiliary LLM's judgment; no formal guarantees on coverage or precision.
  • MACHIAVELLI evaluation is only qualitative/illustrative due to the deterministic environment; no aggregate quantitative results reported for that benchmark.
  • Evaluated on relatively small LMs (GPT2-XL, OLMo-1B, Llama-3.2-1B); behavior on large frontier models is untested.

Relevance to Agentic AI / LLM Agents

ACF directly targets a core explainability gap in LM agents: how to perform semantically valid "what if" reasoning over an agent's actions in environments where language is the action interface. This is highly relevant to agent safety, interpretability, and auditing — particularly for high-stakes deployments (medicine, law, diplomacy) where understanding counterfactual responsibility and deriving corrective policies matters. The framework connects to causal analysis of agentic behavior and extends prior work on counterfactual policy search from RL to the open-ended, text-driven action spaces characteristic of modern LLM agents. It also provides a principled tool for diagnosing and mitigating bias side-effects from representation-steering interventions in deployed agents.