Skip to content

Prompt Coverage Adequacy

🕒 Published (v1): 2026-07-02 11:35 UTC · Source: Arxiv · link

Why this paper was selected

Mark Harman (Meta/UCL); prompt coverage as testing primitive for LLM/agent pipelines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Prompt Coverage Adequacy is a novel test adequacy criterion that measures how well a test suite exercises the natural-language requirements in a prompt, rather than the generated code. It uses LLM attention spotlighting and entropy to detect which prompt requirements are validated by tests. Prompt-coverage-guided test generation reveals 49%/57% of faults on HumanEval+/LiveCodeBench versus only 18%/24% for code-coverage-guided generation.

Problem

Traditional code coverage metrics (statement, branch, mutation) assess test quality against the implementation, not the specification. In LLM-driven development, prompts are the primary artifacts capturing intent. A test suite can achieve 100% statement coverage while entirely missing prompt requirements (e.g., "refund" and "email notification" in a booking system), because the faulty or incomplete LLM-generated code never executes those paths — or the tests are redundant from a code-coverage standpoint yet fault-detecting from a behavioral standpoint.

Method

The criterion operationalizes "requirement coverage" via two ideas grounded in LLM internals:

  1. Entropy as alignment proxy: The cross-entropy \(H_{LLM}(T \mid R)\) measures how surprised the LLM is by test \(T\) given the full prompt \(R\). Higher entropy = lower semantic alignment.
  2. Spotlighting for counterfactual focus: Spotlighting [McDougall et al.] steers attention toward a designated token span \(S\) (a single requirement \(R_i\)) by introducing an additive attention bias \(b_{ij} = \log(\psi^*/\psi_i)\) for \(j \in S\), lifting the relative attention allocation to \(\psi^* = 0.1\).

Coverage criterion: Requirement \(R_i\) is covered by test \(T\) iff

\[H_{LLM}(T \mid \ldots R_i^+ \ldots) - H_{LLM}(T \mid R) < \tau, \quad \tau \leq 0\]

If spotlighting \(R_i\) reduces surprisal, the requirement and test are semantically aligned — \(R_i\) is covered. If it increases surprisal, the forced attention distracts from more relevant context.

Implementation: Entropy is approximated by reducing to a single-token prediction (whether test \(T\) is entailed by \(R\)), avoiding length-dependent entropy accumulation. Prompt segmentation is rule-based (extracting requirement sentences from docstrings). The prompt coverage score is \(PC(\mathcal{T}, R) = |Cov_R(\mathcal{T})| / |R|\).

Evaluated on HumanEval+ (164 tasks) and LiveCodeBench v6 (112 tasks, 2025) using Qwen2.5-Coder (14B) and Gemma3 (12B); test generation guided by prompt coverage vs. statement coverage is compared via fault detection rate.

Key Contributions

  • Definition of Prompt Coverage Adequacy, a prompt-level test adequacy criterion analogous to code coverage but operating on natural-language requirements
  • A concrete instantiation using attention boosting (spotlighting) and entropy change as the coverage signal, requiring no access to source code
  • An efficient single-token entropy approximation to handle variable-length test cases
  • Empirical evidence that prompt coverage is a significant predictor of code coverage (independent of test-suite size) and of fault detection
  • Demonstration that prompt-coverage-driven test generation substantially outperforms code-coverage-driven generation in fault detection across two benchmarks

Results

  • RQ1 (Prompt vs. Code Coverage correlation): Prompt coverage is a strong, significant predictor of code coverage — Qwen2.5-Coder: \(\beta=1.84\), OR \(=6.30\), \(p<0.001\); Gemma3: \(\beta=1.21\), OR \(=3.35\), \(p<0.001\). The interaction with test-subset size is significant but small (OR \(\approx 0.92\)–\(0.93\)), confirming saturation at large suite sizes does not eliminate the PC effect.
  • RQ2 (Prompt coverage vs. fault detection): Higher prompt coverage is a significant independent predictor of fault detection, even controlling for suite size.
  • RQ3 (Guided generation): Prompt Coverage-guided tests detect 49% (HumanEval+) and 57% (LiveCodeBench) of faults; Code Coverage-guided tests detect 18% and 24% — approximately 30+ percentage points more faults with prompt coverage guidance.

Limitations

  • Requires access to model internals (attention weights, logits for entropy); closed-source models are excluded.
  • Spotlighting introduces noise in attention weights, requiring threshold \(\tau\) to be tuned on a held-out training set — this calibration step is not fully automated.
  • Prompt segmentation uses a rule-based parser that may misparse complex or non-standard docstrings.
  • Evaluated only on Python coding benchmarks with well-structured docstring prompts; generalization to prose prompts or multi-file agentic tasks is unvalidated.
  • Coverage quickly saturates in these benchmarks, limiting the dynamic range for RQ1/RQ2 analysis.
  • The entropy approximation (single-token ✓ entailment) discards fine-grained per-token information.

Relevance to Agentic AI / LLM Agents

Agentic and LLM-based software development pipelines treat prompts as the primary specification artifact, yet no principled testing framework has existed at that abstraction level — this work directly fills that gap. Prompt Coverage Adequacy provides a mechanism to validate agent-generated code against the intent expressed in the driving prompt, which is especially critical when the implementation may vary across runs or is unavailable mid-pipeline. The criterion is implementation-agnostic, making it applicable to iterative agentic code generation loops where code structure changes between rounds while the prompt remains fixed. For researchers building or evaluating coding agents, this offers the first specification-aware adequacy metric that can guide automated test generation without requiring ground-truth implementations.