Decision Making Needs Uncertainty Quantification [Lecture Notes]¶

🕒 Published (v1): 2026-07-15 22:50 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This lecture note derives, from first principles within a unified decision-theoretic framework, which form of uncertainty representation is sufficient for optimal action under different agent objectives and knowledge conditions. It unifies calibration, conformal prediction, credal sets with distributionally robust optimization (DRO), and Bayesian inference as complementary answers to one question: what is a sufficient, trustworthy summary of uncertainty for a given decision problem?

Problem¶

Signal processing and AI systems ultimately produce actions, yet uncertainty is typically quantified independently of how it will be used for decision-making. Existing treatments of calibration, robust optimization, and Bayesian inference are presented as separate topics, leaving unclear which representation is necessary and sufficient under each combination of risk attitude (risk-neutral vs. risk-averse) and knowledge profile (known environment, fixed predictor, finite data, parametric model).

Method¶

The paper formalizes a single Bayesian network decision problem with state \(s\), observation \(o\), policy \(\pi(a|o)\), and utility \(u(s,a)\). It then analyzes five settings by deriving the minimal sufficient statistic mapping \(o\) to a decision interface:

Known environment, risk-neutral: The posterior \(p(s|o)\) is the sufficient statistic; the agent maximizes \(U(a|o) = \mathbb{E}_{s \sim p(s|o)}[u(s,a)]\).
Known environment, risk-averse: The agent maximizes value-at-risk \(V_\alpha(a|o)\) (the lower \(\alpha\)-quantile of utility under the posterior). Via Lemma 1 and Proposition 1, this is equivalent to a max–min policy over a level-\(\alpha\) prediction set \(C_\alpha(o)\) satisfying \(\Pr_{s \sim p(s|o)}[s \in C_\alpha(o)] \geq 1-\alpha\), with no loss of optimality for the optimally constructed set.
Unknown environment, fixed predictor: A fixed predictor \(\hat{p}(s|o)\) is optimal (on average over observations) if and only if it is distribution calibrated (Proposition 2): within all observations mapped to the same predictive distribution \(q(s)\), the true conditional frequencies equal \(q(s)\).
Unknown environment, non-parametric data: Given \(D = \{(o_i, s_i)\}_{i=1}^n\), the histogram \(\hat{p}^{\text{hist}}(s|o)\) is an optimistic plug-in. A credal set \(\mathcal{P}(D_o) = \{q(s) : \mathcal{D}(q(s), \hat{p}^{\text{hist}}(s|o)) \leq r\}\) covering the true posterior with probability \(\geq 1-\delta\) yields a DRO action \(a^{mm}(D_o) = \arg\max_a \min_{q \in \mathcal{P}(D_o)} \mathbb{E}_{s \sim q}[u(s,a)]\) whose worst-case certificate \(U^{mm}(D_o)\) satisfies out-of-sample disappointment \(\leq \delta\) (Proposition 3). The KL divergence is asymptotically optimal for the radius choice.
Unknown environment, parametric: Bayesian inference over \(\theta\) yields the parameter posterior \(p(\theta|D)\) and predictive \(p(s|o,D) = \mathbb{E}_{\theta \sim p(\theta|D)}[p(s|o,\theta)]\), which plays the same role as the known-environment posterior.

Key Contributions¶

Unified decision-theoretic framework placing calibration, conformal prediction, credal sets/DRO, and Bayesian inference inside a single problem.
Formal proof that prediction sets with coverage guarantees are the lossless sufficient interface for risk-averse agents (Proposition 1).
Characterization of distribution calibration as the necessary and sufficient condition for a fixed predictor's plug-in policy to be average-optimal (Proposition 2).
Formal proof that DRO over a covering credal set yields a high-confidence performance certificate with controlled out-of-sample disappointment (Proposition 3).
Identification that KL-based DRO is asymptotically least conservative, with disappointment decaying as \(\exp(-n_o r)\).
Critique of ECE as a biased, inconsistent calibration metric; recommendation of binning-free alternatives with rigorous \(p\)-values.

Results¶

No empirical benchmarks are presented; this is a theoretical/tutorial paper. All results are analytic proofs.
Proposition 1: The max–min action over the optimally chosen \(C_\alpha^*(o)\) exactly recovers the value-at-risk optimal action \(a_\alpha^*(o)\) — zero optimality gap.
Proposition 2: Calibrated plug-in policy achieves \(a^*(\hat{p}) \in \arg\max_{a(\hat{p})} \mathbb{E}_{o}[U(a(\hat{p})|o)]\) — optimality among policies using only the predictor output.
Proposition 3: With TV-ball radius \(r = \sqrt{(|S|\log 2 + \log(1/\delta))/(2n_o)}\), the credal set covers the true posterior w.p. \(\geq 1-\delta\), and \(\Pr[U(a^{mm}|o) < U^{mm}(D_o)] \leq \delta\).

Limitations¶

Scope is tutorial/expository; no novel empirical results or experimental validation.
Credal-set and DRO treatment assumes discrete, finite \(|S|\) and \(|O|\); continuous-state extensions are not derived.
Distribution calibration (condition 23) is acknowledged as practically unverifiable in high dimensions due to the need to bin the space of predictive distributions.
Only single-step (one-shot) decision problems are treated; sequential/multi-step settings (MDPs, agentic loops) are not analyzed.
Approximate Bayesian inference methods (MCMC, VI, Laplace) are mentioned but not analyzed for their impact on decision quality.

Relevance to Agentic AI / LLM Agents¶

This note provides rigorous theoretical grounding for a problem central to LLM-based agents: when and how uncertainty about the environment state should be quantified and communicated to a decision-making agent. The calibration result (Proposition 2) directly informs debates about whether LLM confidence scores are trustworthy inputs to downstream planners, and the DRO framework formalizes how agents operating with learned, imperfect world models can still obtain certified performance guarantees. The extension to goal-conditioned agents (Remark 1) maps cleanly onto instruction-following agents where the goal \(g\) modulates both the policy and utility. As agentic systems increasingly involve risk-sensitive deployment (autonomous coding, robotic control, medical decision support), the unified treatment of risk-aversion via prediction sets and value-at-risk offers a principled alternative to ad hoc confidence thresholding.