Toward Auditable AI Scientists: A Hypothesis Evolution Protocol for LLM Agents¶

🕒 Published (v1): 2026-07-10 08:39 UTC · Source: Arxiv · link

Why this paper was selected

Hypothesis evolution protocol for auditable LLM-agent science; addresses harness-level auditability

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

The Hypothesis Evolution Protocol (HEP) is an agent harness that externalizes hypothesis management—proposal, testing, belief updating, and lifecycle resolution—as auditable, persistent operations on top of any LLM agent. Applied to materials-science research tasks, a HEP-equipped agent demonstrably executes the full hypothesis–test–evidence–belief cycle that planning-style agents leave unexercised, and protocol utilization depth scales with base-LLM capability.

Problem¶

In existing LLM agents for scientific discovery, hypotheses and belief updates are buried in unstructured logs; a large-scale trace analysis found evidence was ignored in ~2/3 of traces and beliefs were revised after refutation in only ~1/4 of cases. No mechanism lets the agent or human researcher audit which hypotheses were entertained, which tests were run, or how evidence moved beliefs—making the scientific soundness of agent conclusions unverifiable and preventing systematic hypothesis evolution over long research horizons.

Method¶

HEP wraps a standard LLM agent with two components:

HEP Registry: An append-only, event-sourced, hash-chained store where each hypothesis is a persistent object carrying: a natural-language statement, a belief probability \(P(H)\), a lifecycle state (proposed → under_test → supported/refuted/dormant), lineage (parent hypotheses and generation mechanism), and a testable predicted observable. Every belief update and state transition appends to the hypothesis's event log.
HEP Tools (seven): propose_hypothesis, transition_hypothesis, attach_evidence, list_hypotheses, get_hypothesis, refine_hypothesis, merge_hypotheses. Tools are grouped as Propose & Evolve, Test & Judge, and Read.

Belief is elicited (agent-stated with rationale), not computed. Two hard rules are enforced by the tools: (a) \(P(H)\) moves only via evidence that passes a validation/certification gate; (b) verdict transitions are threshold-gated—supported requires \(P(H) \geq 0.8\), refuted requires \(P(H) \leq 0.2\). Evolution is evidence-driven: refine_hypothesis and merge_hypotheses require parent hypotheses to have reached a verdict or have at least one evidence attachment, preventing untested proposals from generating descendants. The baseline is a plan–execute–replan agent with identical tools and base LLM.

Key Contributions¶

A formally defined Hypothesis Evolution Protocol with a five-state lifecycle, four generation mechanisms (de-novo, inspired-by, refine, merge), and append-only auditable registry
Empirical demonstration that HEP drives the full H→T→E→B→J→U cycle (key transitions: H→T 0.85, T→E 0.58, E→B 0.78, B→J 0.40) while a planning-style baseline spends 83% of steps on tests with 0% on belief updates
Generalization evidence across three structurally distinct materials-science tasks (binary AO2, quaternary A2BB′O6, binary MX), each producing a verified governing rule with final \(P(H) \in \{0.90, 0.96, 0.97\}\)
Scaling characterization: mean hypothesis count drops 14.7→6.7→4.0 and mean generation depth falls 4.7→1.7→0.7 across GPT-5.5→GPT-5.4-mini→GPT-4.1, showing harness exploitation scales with LLM capability
Distinction between harness role (externalizing/structuring reasoning) and capability role (determining reasoning quality), framing them as complementary rather than substitutable

Results¶

Cycle coverage: HEP agent transitions H→T (0.85), T→E (0.58), E→B (0.78); planning baseline: H→T (1.00), T→E (0.61), E→B (0.00)—belief updates never occurred spontaneously in planning-style runs
Hypothesis resolution: All hypotheses reached terminal states (supported/refuted/dormant) in all three tasks under HEP with GPT-5.5; GPT-5.4-mini left 36% open on average; GPT-4.1 was unstable (one of three runs abandoned all hypotheses without attaching evidence)
Final beliefs: AO2 governing rule \(P(H)=0.97\); A2BB′O6 \(P(H)=0.96\); MX \(P(H)=0.90\)
Hypothesis counts per run: AO2 generated 16 hypotheses (example run), tasks overall 10–20 per task; maximum generation depth 3–5 across tasks
LLM scaling: de-novo mechanism share rises from 25% (GPT-5.5) to 53% (GPT-5.4-mini) to 62% (GPT-4.1), indicating weaker models cannot exploit refinement/merge mechanisms
Belief flips: Prior and final belief distributions overlap by final state at proposal but separate cleanly at verdict; flips occur in both directions; no hypothesis proposed above \(P(H)=0.7\) prior was ultimately refuted

Limitations¶

Belief is self-reported and self-assessed by the agent—no programmatic or independent validation of evidence quality; uncertified attachments are recorded but belief is unchanged
Evidence consists of MLIP-level (machine-learned interatomic potential) energetics, not first-principles calculations; correspondence of discovered rules to experimentally established materials physics is unexamined
Single-agent setting only; no independent auditor agent for evidence validation
Evaluated on one task family (materials-science polymorph/ordering questions); generality to other scientific domains is undemonstrated
LLM capability is a hard prerequisite for full protocol exploitation; GPT-4.1-class models struggle to use refinement and merge mechanisms

Relevance to Harnesses / Meta-Harnesses¶

HEP is a textbook example of a domain-specific agent harness: it adds a structured protocol layer above a generic LLM agent without modifying the base model, constraining and externalizing reasoning through tool-mediated state machines rather than prompt engineering alone. The append-only, event-sourced registry design—where state is derived by replaying event logs—is a harness architectural pattern directly applicable to meta-harnesses that need auditable, replayable agent traces. The finding that harness and LLM capability are complementary (harness determines whether reasoning is externalized and verifiable; capability determines how fully it is exploited) is a key design principle for any harness targeting capable models. The paper also foreshadows multi-agent harness architectures (external auditor agent, programmatic validation hooks) as natural extensions of the persistent-registry pattern.