Are Large Language Models Sensitive to the Motives Behind Communication?

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper systematically evaluates whether LLMs can exercise "motivational vigilance"—discounting information from biased or self-interested sources. Frontier non-reasoning LLMs perform near-rationally in controlled vignette settings (Pearson r > 0.78 with a Bayesian normative model) but fail to generalize this vigilance to naturalistic, noisy inputs like YouTube sponsorship transcripts (r < 0.2). A simple prompt steer targeting speaker intentions and incentives partially recovers performance.

Problem

LLMs acting as agents process vast amounts of motivated communication (ads, sponsored content, partisan posts) but lack a principled evaluation of whether they can detect and discount self-interested or biased sources. Prior work identified related failure modes (jailbreaking, sycophancy, susceptibility to pop-up distractors) but provided no unified, normative framework for measuring motivational vigilance specifically.

Method

Three experiments using a Bayesian rational model from cognitive science (Oktar et al.) as a normative benchmark:

1. Experiment 1 (deliberate vs. incidental): Adapted two-player circle-counting task where Player 2 (LLM) receives either deliberate advice or secretly observed answers from Player 1. Measures whether LLMs differentially weight motivated vs. incidental information.

2. Experiment 2 (controlled vigilance): Vignette paradigm across finance, real estate, and medicine domains. Speaker benevolence (λ, via social closeness) and financial incentives (RS, \(0–\)1,000) are varied systematically across 16 speaker-incentive combinations. LLM influence scores are correlated against the rational model's posterior predictions using elicited trust and incentive priors. Evaluated across direct/CoT prompting and first-person/assistant roles.

3. Experiment 3 (ecological validity): 300 real YouTube sponsorship transcripts (SponsorBlock dataset), brand names censored via GPT-4o. Same elicitation of product quality, trustworthiness, and incentive scores. Tests whether internal consistency (correlation with Bayesian model) holds under naturalistic noise. A vigilance-steering prompt ("Consider the motivations... specifically their intentions and incentives") is tested as an intervention.

Key Contributions

• First systematic, normative evaluation of motivational vigilance in LLMs using a Bayesian rational model as benchmark.
• Finding that frontier non-reasoning LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Flash, Llama 3.3-70B) are internally consistent with the rational model in controlled settings (r ≈ 0.79–0.91) and more humanlike than the model predicts.
• Finding that reasoning models (o1, o3-mini, DeepSeek-R1) are paradoxically less vigilant (r ≈ 0.33–0.72), especially in assistant roles (DeepSeek-R1 drops to r = −0.141).
• Demonstration that vigilance collapses in naturalistic settings (r < 0.2) and degrades with transcript length.
• Prompt steering intervention that recovers partial vigilance by making speaker incentives salient.
• A proposed taxonomy (inputs/processes/outputs) to structure future vigilance research.

Results

• Experiment 1: Both GPT-4o and Claude 3.5 Sonnet shift answers more for spied (incidental) than advised (motivated) information, consistent with human data; cooperative payoffs increase shifts.
• Experiment 2 (Bayesian–LLM correlation):
• GPT-4o: r = 0.911; Claude 3.5 Sonnet: r = 0.845; Llama 3.3-70B: r = 0.876
• Reasoning models: o1: r = 0.705; o3-mini: r = 0.716; DeepSeek-R1: r = 0.326
• Small models: Llama 3.2-3B: r = 0.349; Gemma 3-4B: r = 0.288
• Experiment 3 (YouTube sponsorships, default prompt): All models show r < 0.2 with Bayesian model across all conditions; several near 0 or negative.
• Steering intervention: Statistically significant improvements in majority of conditions; e.g., GPT-4o Direct User: r = −0.006 → 0.312; Claude Direct User: r = 0.119 → 0.283.
• Transcript length: Shortest quartile consistently yields higher Bayesian fit than longest quartile across all models and prompting conditions.

Limitations

• Ecological validity experiment relies on a single elicitation per sponsorship (n=1, temperature=0), limiting noise estimation.
• Steering prompt improves but does not close the gap to controlled-setting performance; residual r values remain well below controlled-experiment levels.
• Reasoning models evaluated only n=10 times per condition (vs. 40) due to cost, reducing statistical power.
• Rational model used as benchmark does not incorporate competence vigilance, only motivational vigilance, potentially missing a component LLMs actually track.
• Confounds from LLMs' world-knowledge about specific products/channels are only partially controlled by brand-name censoring.
• CoT prompting increases trust in social information beyond human norms (>60% shifts), a failure mode not explained by the rational model.

Relevance to Agentic AI / LLM Agents

Motivational vigilance is a direct safety-critical capability for autonomous agents browsing the web, processing search results, or acting on third-party recommendations—environments saturated with financially and ideologically motivated content. The finding that reasoning models are less vigilant than standard frontier models, particularly in assistant roles, has immediate implications for agentic deployments that rely on extended chain-of-thought. The performance collapse on naturalistic inputs (YouTube ads) combined with the effectiveness of a simple prompt steer suggests that agent system prompts should explicitly cue source-intent reasoning as a practical near-term mitigation. This work connects to prompt injection and adversarial context research by identifying a behavioral mechanism—failure to ground quality judgments in source incentives when context is noisy—that explains susceptibility to motivated manipulation.