Strategic Hypothesis Testing¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper formalizes statistical hypothesis testing as a Bayesian Stackelberg game between a regulator (principal) who sets a p-value threshold $\alpha$ and self-interested agents (e.g., pharmaceutical firms) who choose whether to participate and how many samples to collect. Despite complex strategic interactions, the principal's Type I and Type II errors are monotone within regions segmented by a critical, efficiently-computable threshold $\hat{\alpha}$. The framework is validated empirically against FDA drug approval data, where the conventional $\alpha = 0.05$ aligns with median revenue thresholds across three drug categories.

Problem¶

Standard hypothesis testing assumes data providers are passive, but in regulatory settings agents are rational: they choose participation and trial size based on cost-benefit calculus. Prior work (Bates et al. 2022) assumed binary product effectiveness and fixed trial costs, precluding strategic sample-size choice and ignoring Type II (false negative) error — which matters for low-revenue and orphan drugs. The resulting non-monotone relationship between $\alpha$ and error rates under strategic behavior makes optimal threshold selection intractable without structural results.

Method¶

The game is defined by the tuple $\mathcal{I} = (q, R, c, c_0, \mu_b)$ where effectiveness $\mu_0 \sim q$ is private to the agent:

Agent action: choose $n \in [n_{\min}, n_{\max}]$ or opt out; cost $= \mathbf{1}[n \neq 0](c_0 + cn)$, revenue $R$ if approved.
Pass probability (normal approximation to binomial): $$\text{Pass}(\alpha, \mu_0, n) \approx 1 - \Phi\!\left(\frac{z_{\alpha,n} - n\mu_0}{\sqrt{n\mu_0(1-\mu_0)}}\right)$$
Principal loss: $L(\alpha, \mathcal{I}) = \lambda_{fp}\,\text{FP}(\alpha,q) + \lambda_{fn}\,\text{FN}(\alpha,q)$ where errors are taken over best-responding agents.

Key structural results derived: 1. Agent best-response (participation + optimal $n$) computable in $O(\log n_{\max})$ via convexity/concavity partitioning and binary search (Theorem 3.1). 2. A participation threshold $\mu_\tau(\alpha)$ — the minimum effectiveness that induces participation — is monotone decreasing in $\alpha$ and computable with $\varepsilon$-precision in $O(\log(1/\varepsilon)\log n_{\max})$ (Lemma 3.2). 3. Critical p-value $\hat{\alpha}$ defined by $\mu_\tau(\hat{\alpha}) = \mu_b$: for $\alpha < \hat{\alpha}$ only effective agents participate (FP = 0); for $\alpha \geq \hat{\alpha}$ ineffective agents enter (FP non-decreasing). Combined FN is non-increasing within each segment. Together this gives a piecewise-monotone, tractably optimizable loss surface.

Empirical validation fits the model to publicly available FDA approval data across oncology, cardiovascular, and infectious disease drug categories.

Key Contributions¶

Stackelberg game model of hypothesis testing with continuous effectiveness, strategic sample-size choice, and marginal per-sample cost — extending Bates et al. beyond binary effectiveness and fixed costs.
Proof that a critical threshold $\hat{\alpha}$ (distribution-agnostic, efficiently computable) cleanly segments error monotonicity: $\text{FP} = 0$ below $\hat{\alpha}$, non-decreasing above; combined $\text{FN}$ non-increasing within each region.
Closed-form characterization of the Stackelberg equilibrium $\alpha^* = \arg\min_\alpha L(\alpha, \mathcal{I})$ via the piecewise-monotone structure.
Empirical validation showing the standard $\alpha = 0.05$ is consistent with median revenue thresholds for three major FDA drug categories under the model.

Results¶

$\alpha = 0.05$ aligns with model-predicted optimal thresholds at median revenue levels across oncology, cardiovascular, and infectious disease drug approval datasets.
Agent best-response computable in $O(\log n_{\max})$; participation threshold with $\varepsilon$-precision in $O(\log(1/\varepsilon)\log n_{\max})$.
Critical threshold $\hat{\alpha}$ is independent of the effectiveness distribution $q$ and the loss scaling parameters $(\lambda_{fp}, \lambda_{fn})$.
FP is identically 0 for all $\alpha < \hat{\alpha}$ (no ineffective agents participate); once $\alpha \geq \hat{\alpha}$, FP grows monotonically.

Limitations¶

Normal approximation to the binomial used throughout; accuracy degrades for small $n$.
Model simplifies the FDA's actual two-trial requirement to a single trial.
Revenue $R$ and cost parameters $(c, c_0)$ are assumed publicly known to the principal; in practice these are private information.
Bernoulli outcomes only; richer multi-dimensional or continuous outcome distributions are not considered.
Strategic behavior restricted to participation and sample-size decisions; direct data manipulation (p-hacking, selective reporting) is outside the model.

Relevance to Harnesses / Meta-Harnesses¶

This paper formalizes the structural analogue of Goodhart's Law in evaluation design: when a meta-harness sets an acceptance threshold, rational agents downstream optimize against it, distorting the error profile the harness was designed to control. The principal-agent Stackelberg framework directly models the adversarial relationship between an evaluation harness (principal) and benchmark-gaming models or pipelines (agents). The key result — that a single critical threshold $\hat{\alpha}$ governs whether gaming introduces false positives — offers a concrete principle for threshold design in automated AI evaluation pipelines: setting $\alpha < \hat{\alpha}$ guarantees zero false approvals even under strategic optimization. For meta-harness designers tracking Goodhart dynamics, the piecewise-monotone structure also provides a tractable basis for principled threshold tuning rather than arbitrary selection.