Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Search

🕒 Published (v1): 2026-07-16 17:48 UTC · Source: Arxiv · link

Why this paper was selected

Static retrieval utility ≠ causal utility in multi-step search; redefines how to build agentic RAG

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Static retrieval utility (does this document help a stateless reader answer the question?) and causal utility (does removing this document hurt the agent's trajectory?) are nearly statistically independent in multi-step agentic search (Spearman \(\rho = -0.026\) over 23,322 document observations). Roughly 35.7% of documents agents read score low on static utility yet are causally load-bearing — the paper calls these "bridge documents." The mechanism is entity propagation: bridge documents supply discriminative entities that redirect the agent's next query.

Problem

Standard retrieval metrics (nDCG, MAP, MRR) and RAG evaluation assume a document is useful if it helps a stateless reader answer the question in one shot. This assumption breaks down for ReAct-style agents that issue multiple queries and reason across turns: a document can be essential without containing any part of the answer, because its value lies in what it enables the agent to search for next. No prior work directly measures the size and mechanism of this static/causal utility gap.

Method

The authors build Counterfactual Trajectory Exploration (CTE) on top of a ReAct agent (Qwen2.5-7B-Instruct + BM25 + cross-encoder reranker) over a stratified 1,000-question sample from HotpotQA dev.

For every document \(d_i\) shown to the agent at step \(t\), they apply an omission intervention: $\([d_1, \ldots, d_{10}] \to [d_1, \ldots, d_{i-1}, d_{i+1}, \ldots, d_{10}]\)$ and replay the trajectory from that point with everything else held fixed (deterministic decoding, temperature 0.0).

The resulting Counterfactual Trajectory Utility (CTU) combines three deltas: $\(\text{CTU} = \tilde{\Delta}_{\text{answer}} + \tilde{\Delta}_{\text{next-query}} + \tilde{\Delta}_{\text{effort}}\)$ where \(\Delta_{\text{answer}} = \text{F1}_{\text{base}} - \text{F1}_{\text{cf}}\), \(\Delta_{\text{next-query}} = \text{nDCG@10}(q_{t+1}^{\text{base}}) - \text{nDCG@10}(q_{t+1}^{\text{cf}})\), and \(\Delta_{\text{effort}} = \text{turns}^{\text{cf}} - \text{turns}^{\text{base}}\). The zero-effect point normalizes to CTU = 1.400, giving the high/low split a principled rather than arbitrary threshold.

CTU is crossed against Static RAG Utility (SRU) — the F1 improvement a fresh, history-free reader gains from the document over its zero-shot parametric baseline — to produce a 2×2 quadrant:

Cell	SRU	CTU	Interpretation
A	high	high	Expected useful
B	high	low	Redundant evidence
C	low	high	Bridge evidence
D	low	low	Genuinely useless

A second experiment tests the mechanism using Observable Entity Relevance (OER) — a support-weighted log-odds measure of whether an entity's presence discriminates relevant from non-relevant candidates — to check whether the entities bridge documents carry are the ones that propagate into the agent's next query.

Key Contributions

• Formal definition of Counterfactual Trajectory Exploration and the omission intervention for isolating a single document's causal contribution in multi-step retrieval.
• CTU metric whose threshold (\(= 1.400\)) coincides exactly with the zero causal-effect point by construction, not by median choice.
• Empirical quadrant experiment over 23,322 document observations showing \(\rho = -0.026\) between SRU and CTU.
• Robustness check replacing the reader-based SRU axis with BM25 + cross-encoder proxies, yielding a bridge cell of 27.2%.
• Mechanistic account: high-OER (discriminative) entities appear in the agent's next query 4.02× more often than low-OER entities (6.1% vs. 1.5%, \(n = 227{,}139\)).
• Pre-registered thresholds (bridge cell \(\geq 20\%\), propagation ratio \(\geq 2.5\times\)) stated before measurement.

Results

• Bridge cell size: 35.7% of all agent-read documents are bridge evidence (Cell C: low SRU, high CTU) under the reader-based SRU axis.
• Rank correlation: Spearman \(\rho = -0.026\) between SRU and CTU — effectively zero, indicating near-independence.
• Robustness (BM25 + cross-encoder proxy): bridge cell = 27.2% with an evenly spread axis distribution.
• Entity propagation ratio: high-OER entities propagate into the next query 4.02× more often than low-OER entities (6.1% vs. 1.5%) over 227,139 entity observations, supporting H1.
• H2 (bridge link): discriminative entities from bridge documents propagate more than those from Cell A documents (exact number not fully quoted in the provided excerpt, but H2 is reported as supported).
• Empirical CTU median = 1.400, coinciding exactly with the theoretical zero-effect point.

Limitations

• All measurements are agent-specific: CTU is formally \(\text{CTU}(D_t, A, t)\) and the authors explicitly make no claim of transfer to other agents or architectures.
• Single agent backbone (Qwen2.5-7B-Instruct); behavior of larger or instruction-tuned models may differ.
• Coverage filter discards questions whose supporting facts are unreachable in the top-50 BM25 candidates, potentially biasing the sample toward questions where retrieval is already functional.
• Skew in the static SRU axis (most documents cluster at low SRU) constrains how much the headline quadrant percentage can bear on its own, as acknowledged by the authors.
• OER entity linking uses WAT with a confidence threshold of \(\rho \geq 0.1\); linking errors could inflate or deflate propagation rates.
• Counterfactual replay holds the prefix fixed but cannot control for in-weights model knowledge, so some trajectories may recover despite deletion via parametric memory.

Relevance to Agentic AI / LLM Agents

This paper directly challenges the evaluation foundations of retrieval-augmented agents: the standard proxy used to train and rank retrieval components (static reader utility) is shown to be nearly orthogonal to what actually matters for multi-step agent success. For researchers building or evaluating LLM agents that use tool-augmented search (ReAct, OpenAI function-calling search loops, agentic RAG pipelines), this means optimizing retrievers on standard IR metrics may leave a large fraction of causally essential evidence systematically undervalued. The bridge document concept and CTU metric provide a concrete, operationalizable alternative: evaluate documents not by what they contain relative to the current query, but by what trajectory degradation their removal causes — a direction that points toward counterfactual or causal training signals for next-generation retrieval components in agentic settings.