Skip to content

Many LLMs Are More Utilitarian Than One

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multi-agent LLM systems exhibit a "Utilitarian Boost" — a systematic shift toward endorsing norm-violating actions that maximize collective welfare — when identical agents deliberate in groups, replicating a known human group phenomenon but via different mechanisms. The effect was observed across all six tested LLMs (GPT-4.1, Llama3.3, Gemma3, Qwen2.5, Qwen3, QwQ) in personal dilemmas (direct harm), but not impersonal ones, the reverse of the human pattern. Prompt diversity, architectural heterogeneity, and mixed-capacity pairings can dampen or reverse the boost.

Problem

Prior LLM moral reasoning research evaluates single agents in isolation, leaving a blind spot: do multi-agent systems produce emergent collective moral biases not present in any individual agent? No prior work systematically characterized whether and how group deliberation among LLMs shifts moral judgment, what mechanisms drive it, or how MAS design choices can modulate it.

Method

Six LLMs (five open-source + GPT-4.1, including two reasoning models QwQ/Qwen3) evaluated moral dilemmas in two conditions: Solo (independent judgment) and Group (multi-turn structured dialogue in dyads or triads over 6 rounds, followed by private reflection). Moral acceptability was rated on a 1–7 scale (higher = more utilitarian). Statistical analysis used cumulative mixed-effects ordinal regression with random intercepts/slopes for dilemma variability. Three instrument batteries measured the boost and its mechanisms: (i) Greene's classical personal/impersonal sacrificial dilemmas for overall Utilitarian Boost; (ii) Oxford Utilitarianism Scale (OUS) separating Impartial Beneficence from Instrumental Harm; (iii) CNI model deriving latent sensitivity to Consequences (C), Norms (N), and Inaction preference (I). Post-hoc experiments varied role prompts (deontological/utilitarian/neutral), model family heterogeneity, and capacity heterogeneity (strong vs. weak model pairing). Human crowd-sourced ratings on Prolific validated argument coherence and score alignment.

Key Contributions

  • First systematic demonstration of a Utilitarian Boost in LLM-MAS across six diverse models and two group sizes, significant in all cases (overall β = 0.31, z = 6.81, p < .0001).
  • Shows the boost is specific to personal (direct-harm) dilemmas, absent in impersonal ones — the inverse of the human pattern, which is stronger for impersonal harm.
  • Reveals model-specific mechanistic profiles via CNI: Gemma3 = norm-aligned optimizer (reduced N); GPT-4.1 and Llama3.3 = impartial utilitarian (increased IB); Qwen3 = action-focused utilitarian; contrasting with humans where only enhanced consequence sensitivity (C) drives the boost.
  • Identifies affective language shift (solo: disgust-dominant → group: fear-dominant) as a correlate of the boost, paralleling human emotion-mediation theories.
  • Provides design levers to modulate the boost: mixed moral-framing prompts (UD/DU dyads produce a Deontological Boost, Joint − Round1 = −0.323, p < .0001), cross-family model pairing (β = −0.30, p = .0001), and capacity-heterogeneous pairing (reverses boost, β = 1.40, p < .001).

Results

  • Overall Utilitarian Boost: β = 0.31, SE = 0.046, z = 6.81, p < .0001; holds for pairs (s=2) and triads (s=3).
  • By model (Group vs. Solo estimates): Gemma3 (β = 1.65, p < .0001), Qwen3 (β = 1.23, p < .0001), Llama3.3 (β = 0.80, p < .0001), QwQ (β = 0.69, p < .0001), Qwen2.5 (β = 0.68, p < .0001), GPT4.1 (β = 0.57, p = .0023, only in larger groups).
  • Personal dilemmas: Strong boost (β = 0.64, z = 14.31, p < .001); Impersonal dilemmas: No boost (β = −0.023, p = .975).
  • Mitigation — prompt diversity: Homogeneous DD dyads still show boost (+0.377, p = .010); mixed UD/DU dyads produce Deontological Boost (−0.323, p < .0001).
  • Mitigation — capacity heterogeneity: Mixed strong/weak pairing reverses the boost (β = 1.40, SE = 0.17, z = 8.28, p < .001).
  • Mitigation — architectural heterogeneity: Cross-family pairs dampen boost (β = −0.30, p = .0001) vs. homogeneous pairs still boost vs. solo (β = +0.29, p = .0001).
  • Utilitarian agents conform more: Agents starting utilitarian are more likely to shift toward deontological than vice versa (Δ log odds = −0.467, p < 10⁻⁴).

Limitations

  • Only dyads and triads studied; large panels, committee topologies, and asynchronous deliberation untested.
  • Meta-controllers (turn schedulers, position summarizers) could modulate results in unpredictable ways.
  • Experiments primarily in English, training distributions skewed Western; cross-lingual and cross-cultural generalization unvalidated.
  • Affective language findings are correlational and classifier-dependent (two pretrained models used).
  • Post-hoc mitigation experiments are exploratory, not pre-registered.

Relevance to Agentic AI / LLM Agents

This paper directly challenges a core assumption in multi-agent LLM design: that aligning individual agents is sufficient to guarantee aligned collective behavior. The emergent Utilitarian Boost constitutes a safety-relevant alignment failure mode unique to MAS interaction, invisible to single-agent benchmarks. The finding that group composition (homogeneous vs. heterogeneous models, prompt framing) functionally controls moral drift provides actionable guidance for practitioners designing multi-agent pipelines in high-stakes domains. It also highlights a critical gap in current agent evaluation infrastructure: collective moral reasoning benchmarks do not yet exist, meaning deployed systems in healthcare, law, or policy advisory settings may harbor untested group-level biases.