Set-shifting Behavioral Test for Harnessed Agents¶

🕒 Published (v1): 2026-07-15 02:49 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper adapts the Wisconsin Card Sorting Test (WCST) from cognitive psychology into a benchmark for LLM agents operating in persistent agentic harnesses: a hidden reliable tool group shifts mid-session and the agent must update its routing using only binary success/failure feedback. Two open-weight models exhibit qualitatively distinct perseveration patterns—one tied to prior use history, one to the first commitment in a session—and both are partially steerable via policy prompts or set framing.

Problem¶

LLM agents in agentic harnesses mount large libraries of redundant tools (functionally equivalent tools differing in hidden reliability). Agents converge on preferred subsets that can persist after a silent backend change, yet no benchmark measures whether agents adapt their tool routing when the reliable group shifts mid-trajectory within an ongoing session. Prior cognitive tests on LLMs use textual/visual sorting without callable tools, harness state, or resumed sessions.

Method¶

A redundant tool-skill set mounts multiple tool groups $C = \{C^{(1)}, \ldots, C^{(M)}\}$ per domain; each group shares identical schemas and descriptions but differs in hidden reliability. Three domains instantiate this: scheduling/coordination ($M=3$, 5 tools/group), DevOps incident triage ($M=4$, 5 tools/group), and multi-cloud storage ($M=5$, 2 tools/group).

A branched schedule tree ($M=3$, $K=2$ layers, 90 turns) places hidden shift boundaries at turns 15, 40, and 65. At every internal node the schedule forks into: reversal to a prior-reliable group, novel shift to a never-reliable group, and a no-shift control. Sibling sets share a parent session state via resumed Hermes Agent sessions (persistent context). The environment emits only binary feedback (success payload or bare failure, no diagnostic message), matching WCST's hidden-rule paradigm.

Set-shifting accuracy is the primary metric: $$\Phi = \prod_{k=1}^{K} s_k$$ where $s_k$ is the share of route calls in post-shift window $P_k$ landing in target group $C_k$. Route classes $R_1$ (target only), $R_2$ (target + one other), $R_3$ (target + both others), $R^*$ (no target) partition each turn.

Two open-weight models (mimo-v2.5, deepseek-v4-pro) are tested via OpenRouter at temperature 0.7. Two interventions are evaluated: policy prompts appended to Hermes's SOUL.md (adaptive vs. polymath personas) and set framing (competing vs. complementary descriptions of the same toolset).

Key Contributions¶

Benchmark of redundant tool-skill sets with branched, paired (shift/no-shift) reliability schedules for harnessed LLM agents
Descriptive framework characterizing post-shift behavior as convergence to a small set of discrete stable routines ($s \in \{0, 0.33, 0.5, 1\}$), interpreted as attractor states
Empirical taxonomy of two qualitatively distinct failure modes across the same harness: cell-level lock-in (mimo, prior-use-dependent partial perseveration) vs. prefix-level lock-in (deepseek, bimodal first-commitment perseveration)
Demonstration that adaptive policy prompts can redirect (but not break) deepseek's lock-in, while set framing shifts routine occupancy across both models

Results¶

mimo-v2.5 ranks $C_A > C_B > C_C$ in $\Phi$ (Tier-1: 0.80, 0.44, 0.17); settles to mid-range routines; prior-use role predicts which attractor it occupies
deepseek-v4-pro shows bimodal trajectories ($s \approx 1$ or $s \approx 0$); Tier-1 $\Phi$: 0.53 ($C_A$), 0.86 ($C_B$), 0.40 ($C_C$); first-turn commitment drives all later routing
Tier-1 shift costs deepseek ~0.40 in $\Phi$ vs. no-shift control ABB (paired permutation: ABA $p=6\times10^{-4}$, ABC $p=6\times10^{-5}$)
Adaptive policy prompt on deepseek: $s \geq 0.86$ at every Tier-2 endpoint ($\sigma \leq 0.06$); eliminates bimodality; ABAC $s$ goes from 0.30 (9/16 trajectories at $s \leq 0.2$) to 0.87 (0/8 below 0.2)
Polymath prompt uses ~2× unique tools per window but does not consistently increase $\Phi$
Set framing: competing framing raises $\Phi$ over complementary in 26/32 pairs (sign test $p=3\times10^{-4}$); complementary framing pins all trajectories to a single mixed routine (e.g., deepseek DevOps converges at $(1/4)^3$)

Limitations¶

Only two open-weight models tested; the main branched schedule is instantiated in the scheduling domain only (DevOps and multi-cloud run linear, unranched paths)
Small sample sizes ($n=16$ prefixes per cell for main experiment; $n=8$ for policy prompt and framing conditions)
Policy prompt interventions applied only to deepseek-v4-pro at scale (mimo tested with $n=3$, no reliable steering found)
Single harness (Hermes Agent); results may not generalize to other scaffolding systems or context-management strategies
Binary-only feedback (no diagnostic message) is ecologically valid but rules out testing richer feedback regimes
The paper is truncated before the conclusion is complete; broader model panel results remain future work

Relevance to Harnesses / Meta-Harnesses¶

Harnesses that route tool calls across persistent sessions are exactly where perseveration bugs land hardest: an agent that locks onto a stale tool group silently degrades without surfacing an error, and the only signal is downstream task failure. This paper gives harness builders a concrete behavioral diagnostic—a WCST-style probe—to characterize a model's flexibility before deploying it in a production scaffold, and the two distinct perseveration patterns (history-anchored vs. first-commitment-anchored) directly inform scaffold design decisions: whether to inject context resets, rotate tool descriptions, or insert reframing prompts at session boundaries. The partial steerability finding is the most actionable result: it means meta-harness policy layers (system-prompt engineering, set framing at tool-group boundaries) can compensate for model-level rigidity without fine-tuning, which is the exact lever a harness designer has available at runtime.