Skip to content

Dropouts in Confidence: Moral Uncertainty in Human-LLM Alignment

🕒 Published (v1): 2025-11-17 12:13 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLMs exhibit overconfident moral decisions that misalign with human preferences; this work measures that overconfidence via binary entropy decomposed into total entropy, conditional entropy, and mutual information across 32 open-source models on trolley-problem scenarios. Injecting stochasticity via attention dropout at inference time increases mutual information and significantly improves human-LLM moral alignment scores.

Problem

LLMs are systematically overconfident in morally ambiguous scenarios, producing decisive outputs that diverge from human moral preferences—which are inherently uncertain and value-conflicted. Prior alignment work measured aggregate agreement but did not quantify or manipulate model-level uncertainty as a lever for improving alignment.

Method

The authors use the Moral Machine framework (trolley problems, 9 moral dimensions) with two datasets: an AlignmentSet (10,000 randomly generated scenarios evaluated against Awad et al. 2018 human AMCE scores) and an UncertaintySet (9,000 scenarios isolated per dimension). Binary choice probabilities are extracted from output logits for tokens "1" and "2" via softmax. Uncertainty is quantified as binary entropy H(p), decomposed into total entropy (TE = H(E[p])), conditional entropy (CE = E[H(p)]), and mutual information (MI = TE − CE). To induce stochasticity, attention dropout (rates r ∈ {0.05, 0.1}) is applied to the attention weight matrix at inference time. Human-LLM alignment is measured as L2 distance between model and human AMCE vectors across 9 dimensions.

Key Contributions

  • First systematic application of information-theoretic uncertainty decomposition (TE/CE/MI) to LLM moral decision-making across 32 models and 9 dimensions.
  • Finding that confidence variance is greater across model architectures than across moral dimensions, implicating training methodology as the primary driver of moral uncertainty.
  • Inference-time attention dropout as a mechanism to deliberately modulate LLM overconfidence without retraining.
  • Empirical demonstration that dropout-induced uncertainty increases MI and improves human-LLM moral alignment (negative ΔL2) for most models, with MI increase as the primary predictor of alignment improvement.
  • Blind human evaluation (n=3 annotators, 100 Q/A pairs on Qwen3-1.7B) confirming subjective alignment improvement post-dropout (MSE: 0.369 → 0.263).

Results

  • Attention dropout (r=0.1) produces statistically significant increases in TE (p=5.2e−11) and MI (p=9.4e−10) while CE remains unchanged (p not significant; Bonferroni corrected).
  • Most of the 32 models show reduced L2 alignment distance under dropout; top improvements include Llama-3.1-70B (ΔL2 = −0.15 at r=0.1), Phi-4 (ΔL2 = −0.20), Qwen3-8B (ΔL2 = −0.46), Qwen3-1.7B (ΔL2 = −0.51), Qwen2.5-3B (ΔL2 = −0.50).
  • Gemma family models show consistently high confidence (low uncertainty) baseline; Llama family shows relatively lower confidence.
  • MI change is the sole uncertainty component significantly correlated with ΔL2; TE and CE changes are not significant predictors.
  • Alignment gains are largely preserved under prompt paraphrasing ("Option A/B" format), confirming robustness to minor prompt variation.

Limitations

  • Dropout is injected ad hoc at inference time; uncertainty should ideally be learned from training data reflecting cultural norms rather than applied post-hoc.
  • Alignment improvement is not universal: dropout degrades alignment on specific dimensions (e.g., Age, Species) for some models.
  • Increased stochasticity raises decision variance, which is undesirable in high-stakes deployment contexts.
  • Study is restricted to open-source models; commercial LLMs (GPT-4, Claude, Gemini) are excluded.
  • The Moral Machine framework itself is culturally biased toward Western preferences (original Awad et al. dataset), limiting generalizability.
  • Small human evaluation (n=3 annotators) for the blind subjective alignment check.

Relevance to Agentic AI / LLM Agents

As LLM agents are increasingly deployed in decision-making pipelines—including ethically sensitive domains such as healthcare triage, legal recommendation, and autonomous vehicles—their overconfidence in morally ambiguous situations is a direct safety concern. This work provides a concrete, model-agnostic intervention (inference-time attention dropout) that reduces overconfidence without retraining, relevant to deploying agents whose outputs must reflect calibrated uncertainty rather than spurious decisiveness. The information-theoretic framing (MI as the driver of alignment) connects to broader agent calibration research and suggests that uncertainty-aware sampling strategies could be a first-class design primitive for agentic systems facing value-laden decisions. The finding that architecture and training method dominate moral uncertainty more than scenario content has implications for model selection and RLHF design in alignment-focused agent frameworks.