Skip to content

A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Consumer choice experiments quantify how LLM agents behave as economic decision-makers

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ABxLab is a man-in-the-middle framework that intercepts real web content to run controlled behavioral experiments on LLM agents acting as consumers. Across 17 state-of-the-art models and 80,000+ trials in a web shopping environment, agents show systematic biases toward ratings, prices, order, and psychological nudges that are 3–10× stronger than human responses to the same stimuli. The work proposes consumer choice as a rigorous testbed for a behavioral science of AI agents.

Problem

Current agent benchmarks measure task competence (completion rate, correct button clicks) but ignore decision quality—how agents choose when faced with ambiguous, realistic options in environments engineered to influence behavior. There is no systematic framework for probing whether LLM agents are manipulable by standard real-world persuasion techniques (price signals, ratings, nudges), a gap that matters because agents increasingly make high-stakes decisions (purchases, travel, medical) on users' behalf.

Method

The ABxLab framework formalizes the agent's environment as \(E = \langle S, A, O, T, I \rangle\), where \(I = \{I : O \to O\}\) is a set of intervention functions that modify observations before they reach the agent, enabling real-time content injection without building custom environments. The benchmark applies this to OneStopMarket (WebArena's shopping domain) using a 2-alternative forced choice (2AFC) paradigm: agents browse two product pages in separate tabs and add one to cart.

Product pairs are filtered for realism via validity constraints: $\(|\text{rating}(p_1) - \text{rating}(p_2)| \leq \Delta_r \quad \text{and} \quad \frac{|\text{price}(p_1) - \text{price}(p_2)|}{\min\{\text{price}(p_1), \text{price}(p_2)\}} \leq \Delta_p\)$

Three matching regimes are tested: Original (unmatched), Matched Ratings (MR, \(\Delta_r=0\)), and Matched Ratings & Prices (MRaP, post-hoc price equalization via intervention). Ten nudge interventions span five categories (authority, social proof, scarcity, negative framing, incentives). Effects are estimated via linear probability models with cluster-robust standard errors. A 30-participant Prolific human baseline uses the identical trial set. Total scale: >80,000 experiments, ~2.5B tokens, 17 models including GPT-5, o3, Claude Sonnet 4, Gemini 2.5 Pro, Llama 4, DeepSeek-R1.

Key Contributions

  • Open-source man-in-the-middle intervention engine that transforms arbitrary websites into controlled behavioral testbeds without requiring custom environment construction.
  • Scalable benchmark covering 17 frontier models, 10 nudge interventions, and three attribute-matching regimes (Original, MR, MRaP).
  • Empirical datasets enabling causal inference about what drives agent choices (ratings, prices, order, nudges).
  • Human behavioral baseline (n=30) for direct human-vs-agent comparison on the same trials.
  • Evidence that LLM agent biases are systematic across model families, not artifacts of individual architectures.

Results

Effect sizes reported as absolute percentage-point (pp) change in selection probability:

  • Human baselines: order 4pp (n.s.), rating 5pp (n.s.), price 9.4pp (n.s.), nudge 9.9pp (p<.05).
  • Ratings: 14/17 models showed 30–80pp sensitivity in the Original condition. Strongest: o4-Mini +81.2pp, o3 +77.6pp. Even modest cases (GPT-4o Mini +20.6pp) are >4× the human baseline.
  • Prices: 13/17 models preferred cheaper options (15–34pp range, Original). When ratings were matched (MR), sensitivity intensified dramatically—Llama 4 Maverick reached 93.2pp. MRaP condition eliminated price effects, confirming agents respond to prices directly.
  • Order effects: GPT-4.1 Nano showed +92.7pp bias toward first-listed item (MRaP); Claude 3.5 Haiku showed −42.7pp against it. In matched conditions 13/17 models showed significant order sensitivity, typically favoring the second-viewed product.
  • Nudges (MRaP condition): Claude Sonnet 4 +55.9pp, GPT-4o +62.1pp, GPT-4.1 Mini +41.5pp. All nudges significant at MRaP. Strongest individual nudge: Wirecutter authority cue, followed by "Buy 1 Get 1 Free," then "best seller."
  • Average attribute sensitivity: humans ~7pp; lowest model Claude 3.5 Haiku ~13pp; highest Claude Sonnet 4 ~31pp.

Limitations

  • Observation space is text-only pruned HTML (no visual input); visual nudges not tested.
  • Binary forced choice (2AFC) may not reflect real multi-product selection behavior.
  • Single domain (online shopping/OneStopMarket); generalizability to other agentic tasks untested.
  • Fixed low temperature (0.1) may understate variance; reasoning models use temperature 1.
  • Human baseline is small (n=30, Prolific) and may not represent diverse real users.
  • Sensitivity analyses (Figure 4) suggest bias magnitude does not scale simply with attribute difference magnitude, but the truncated text limits full interpretation of these results.

Relevance to Agentic AI / LLM Agents

This paper directly attacks a gap in current agent evaluation: task-completion benchmarks (WebArena, VisualWebArena) tell us whether an agent can act but not how well it chooses under ambiguity. The finding that agents are 3–10× more susceptible to standard marketing nudges than humans, without the cognitive constraints (bounded rationality, cognitive load) that produce human biases, challenges the assumption that LLMs make more rational proxies. The man-in-the-middle injection architecture is a generalizable evaluation primitive: it can extend to any web-based environment and any intervention type, offering a principled methodology for pre-deployment behavioral auditing of agents operating in human-designed digital spaces. This work sits at the intersection of alignment, trust, and agentic robustness research.