Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents¶

🕒 Published (v1): 2026-05-30 02:02 UTC · Source: Arxiv · Venue: COLM · link

Why this paper was selected

COLM; measures faithfulness gap between stated LLM reasoning and actual actions

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LLM agents almost always execute what they explicitly state (conclusion→action inconsistency: 0–1.8%), but frequently derive invalid conclusions from their own correctly stated reasoning (reasoning→conclusion). The widely reported ~22–26% "unfaithfulness" rate is largely a measurement artifact of free-text conclusion extraction, not genuine behavioral incoherence.

Problem¶

Prior work on CoT faithfulness conflates two distinct pipeline steps—whether an agent executes its stated decision, and whether that stated decision actually follows from its reasoning—and measures both via free-text phrase parsing, which systematically overstates inconsistency. No controlled, verifiable testbed existed to disentangle these failure modes.

Method¶

The authors build a Texas Hold'em multi-agent simulator (four LLM seats) where every decision has a computable reference action: a fixed equity-based threshold heuristic (\(\text{fold if } e < p\); \(\text{call if } |e - p| < 0.05\); \(\text{raise if } e > p + 0.05\), where \(e\) = Monte Carlo equity over 500 rollouts, \(p\) = pot odds). The decision pipeline is split into two measurable steps:

Reasoning→conclusion: given the agent's own stated equity, pot odds, and rule, does its stated decision follow logically?
Conclusion→action: does the executed JSON action match the explicit DECISION: tag?

Four prompt strategies vary how much decision guidance is externalized (Baseline, CoT, Persona, GTO-constrained). A fifth diagnostic strategy (GTO-verbose) forces the agent to emit labeled EQUITY, POT_ODDS, RULE, and DECISION fields, enabling per-field cross-checking. Conclusion→action consistency is measured with both a free-text phrase parser and an explicit tag parser to quantify extraction noise. Experiments cover Claude Haiku 4.5, Gemini 2.5 Flash-Lite (200 hands each), and DeepSeek-v4-pro (50 hands).

Key Contributions¶

Measurement caution: Explicit DECISION tag elicitation collapses apparent inconsistency from 22–33% (free-text) to 0–1.8%, showing prior "unfaithfulness" reports are dominated by extraction noise rather than model behavior.
Decomposition localizing the gap: The faithfulness problem is concentrated in reasoning→conclusion, not conclusion→action; agents reliably execute what they state.
Characterization of upstream failure: 65% of erroneous decisions are rule misapplications—the agent states correct inputs and the correct rule, then produces a conclusion that contradicts that rule, consistently in a risk-averse direction (97% of misapplications override a "raise" signal with a passive action).
Injecting the rule doesn't help: GTO-constrained prompting consistently ranks at or near the bottom in GTO adherence across all three models (\(\chi^2\) significant for Haiku and Flash-Lite), isolating rule application rather than rule retrieval as the bottleneck.
Portable process-fidelity instrument: The harness and step-level fidelity probe are released as a reproducible testbed applicable to social simulations where approximate reference behavior can be defined.

Results¶

Conclusion→action inconsistency (explicit tag parser): 0.3% (Haiku), 1.8% (Flash-Lite), 0.0% (DeepSeek); free-text parser reports 26.4%, 22.0%, 33.3% respectively—roughly 10× overstated.
GTO adherence (overall, GTO-verbose, Claude Haiku 4.5): 60.6% across 791 decisions.
Reasoning→conclusion error breakdown (164 interpretable failures, Claude Haiku 4.5): 65% rule misapplication, 19% equity miscalculation (\(>0.15\) off Monte Carlo), 16% borderline.
Input accuracy: agent-stated equity is within 0.15 of Monte Carlo truth in 68% of decisions—inputs are mostly correct; the failure is inferential.
Rule misapplication directionality: 97% override a "raise" implication with a passive action; mean stated equity exceeds the raise threshold by 0.22, and by \(>0.10\) in 76% of cases.
GTO-constrained strategy: ranks last or near last in adherence across all three models (47.9–54.6%), confirming rule injection does not improve compliance.

Limitations¶

Three-model scope; DeepSeek run uses only 50 hands, yielding wide confidence intervals.
Reasoning→conclusion analysis (GTO-verbose) run only on Claude Haiku 4.5; cross-model replication is future work.
GTO-verbose elicitation may itself alter behavior relative to unconstrained generation.
Texas Hold'em lacks natural-language interaction, persuasion, and norm formation; the testbed is a calibration environment, not a full social simulation proxy.
The reference heuristic is a simplified threshold rule, not an exact game-theoretic equilibrium (exact multi-player Hold'em equilibria are intractable).

Relevance to Agentic AI / LLM Agents¶

This work directly addresses a foundational concern in LLM agent design: whether chain-of-thought reasoning actually governs agent behavior. By splitting the faithfulness gap into two measurable steps, it shows that the bottleneck for agentic reliability is not action execution but logical inference from self-stated reasoning—agents produce internally inconsistent derivations even when inputs and rules are explicitly provided and approximately correct. For researchers building persona-driven or instruction-following agents, the finding that rule injection fails and that hedging language systematically overrides quantitative rules has direct implications for prompt design, evaluation methodology, and the trustworthiness of emergent multi-agent behaviors in social simulation.