Skip to content

Humans Disengage, Reasoning Models Persist: Separating Difficulty Registration from Deliberation Allocation

🕒 Published (v1): 2026-06-25 01:21 UTC · Source: Arxiv · link

Why this paper was selected

LRMs spend more tokens on wrong answers—opposite to human disengagement; rethinks agent eval

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Large reasoning models (LRMs) and humans both spend more deliberation on harder items (difficulty registration), but diverge sharply in deliberation allocation: LRMs generate longer reasoning traces on trials they get wrong, while humans spend less time on trials they fail. This opposite within-item policy is masked by the cross-item correlation used in prior alignment claims.

Problem

Prior work (de Varda et al., 2025) showed that LRM reasoning-trace length correlates with human reaction time across items, suggesting cognitive alignment. This cross-item "registration" diagnostic cannot detect whether agents allocate more deliberation to their own successes or failures. The paper targets exactly that gap: the within-item deliberation allocation policy is orthogonal to cross-item alignment and has not been tested.

Method

Re-analysis of a public matched human–LRM corpus across three reasoning paradigms: H-ARC (visual abstraction, primary), INTUIT (intuitive physics, replication), and Cortes (binary relational reasoning, boundary). Six open-weight thinking LRMs (DeepSeek-R1, Qwen-QwQ-32B, Qwen3-235B-Thinking, GLM-4.5-Air-FP8, gpt-oss-20b, gpt-oss-120b) plus non-thinking DeepSeek-V3 as control.

Two diagnostics are introduced: 1. Cross-item registration — Spearman \(\rho\) between LRM trace length and human RT across items (replicates prior work). 2. Within-agent allocation — outcome-conditioned Cohen's \(d\) on log deliberation: \(d_{\text{wrong}-\text{right}}(A, P) = (\bar{\ell}_{\text{wrong}} - \bar{\ell}_{\text{right}}) / s_{\text{pooled}}\), with \(\ell_i = \log t_i\).

Item fixed-effects regressions absorb the cross-item difficulty gradient and identify the human–LRM interaction \(\beta_{\text{LRM}}\) from within-item variation. Leave-one-out ensemble difficulty and Mundlak re-parameterisation serve as robustness checks. Trace-content features (self-doubt marker density, 5-gram repetition, type–token ratio) probe the within-system mechanism.

Key Contributions

  • Formally separates difficulty registration (cross-item) from deliberation allocation (within-item), which prior human–LRM alignment work conflated.
  • Introduces an outcome-conditioned, item-controlled diagnostic that operates within each agent's native scale (seconds vs. tokens never directly compared).
  • Shows that the cross-item alignment is real but masks opposite within-item allocation policies: humans disengage on failures; LRMs generate more tokens precisely when they fail.
  • Provides a mechanistic interpretation: human RT is partly gated by engagement/abandonment; LRM length tracks uncertainty-like trace features (hedging markers, lexical contraction).

Results

  • Registration (cross-item): Thinking LRMs replicate de Varda et al.'s finding — Spearman \(\rho = 0.16\)–\(0.30\) (all \(p < .001\)) between trace length and human RT on H-ARC. Non-thinking V3: \(\rho = -0.05\).
  • Allocation (within-item), H-ARC: All well-powered thinking LRMs produce \(d > 0\) (wrong > right), range \(d = 1.47\)–\(3.13\); matched human sample is at \(d = -0.10\).
  • Item fixed-effects primary estimand: \(\beta_{\text{LRM}} = -0.66\) log units (\(p < .001\), 95% CI \([-0.81, -0.50]\), \(n = 5{,}728\)); four sensitivity specifications all strictly below zero (\(-0.66\) to \(-0.94\)).
  • INTUIT replication: Every thinking LRM at \(d = 0.59\)–\(1.42\); human \(d = +0.07\); \(\beta_{\text{LRM}} = -0.41\) (\(p < .001\)).
  • Cortes boundary: Agent-type interaction preserved (\(\beta_{\text{LRM}} = -0.97\), \(p < .001\)) but LRM-only within-item slope reverses sign, treated as a paradigm-dependent boundary case.
  • Non-thinking V3: Positive \(d\) on Cortes (\(d = 1.41\)), directionally positive but underpowered on H-ARC — suggests part of wrong-trial length expansion is shared with non-thinking generation under uncertainty.
  • Trace content: Wrong-trial H-ARC chains have 0.43 more self-doubt/hedging markers per 1,000 characters (\(p = 7 \times 10^{-4}\)); wrong Cortes chains show 7.4% higher 5-gram repetition and 4.1 pp lower type–token ratio — convergent uncertainty signature.
  • Human engagement probe: Adding log(1 + action_count) as covariate halves the human within-item slope from \(+0.24\) to \(+0.12\), consistent with abandonment driving short wrong-trial RT.

Limitations

  • LRM sample is open-weight only; closed-frontier models (GPT-4o, Gemini, Claude) not tested.
  • Public release contains exactly one observation per (item × model) pair — the within-item LRM slope is identified across models, not across repeated stochastic samples of a single model, so it is not a per-model stochastic estimate.
  • Participant identifiers unavailable, so clustering is at item level rather than by person.
  • Cortes has only 5–8 wrong trials per LRM; bootstrap intervals are wide and mechanistic interpretation is withheld.
  • Trace tokens are treated as a behavioral readout of generation policy, not as a meter of latent computation — the degree to which trace length reflects real internal computation remains open.
  • Causal direction between uncertainty and trace length is not established; the study is correlational.

Relevance to Agentic AI / LLM Agents

This paper directly challenges the assumption that LRM "thinking" is analogous to human deliberation, which is foundational to arguments for chain-of-thought as a cognitive scaffold in agentic systems. For LLM agent designers, the key finding is that token budget ≠ problem-solving effort: models spend more tokens precisely on the problems they fail, making trace length a poor proxy for confidence or solution quality. This has immediate implications for stopping criteria, self-evaluation, and compute allocation in agentic loops — an agent that runs longer is not necessarily closer to a correct answer. The dissociation between registration and allocation also provides a cleaner framework for evaluating metacognitive control in agents beyond surface-level reaction-time analogies.