Skip to content

DeepStress: Stress-Testing Deep Search Agents

🕒 Published (v1): 2026-07-15 14:59 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DeepStress is a controlled stress-testing framework that replaces a search agent's retrieval module with a synthetic environment to evaluate robustness under degraded evidence quality. It manipulates three orthogonal document-reliability axes—trustworthiness, relevance, and factuality—and introduces new metrics (RAS, CoPRAS) that reward appropriate abstention under unreliable conditions. Testing 12 agents on HotpotQA and BrowseComp-Plus reveals dramatic, system-specific failure modes invisible to standard accuracy metrics.

Problem

Standard search-agent evaluations collapse performance to final-answer accuracy, masking qualitatively different failure modes (hallucination, abstention, format collapse, tool-budget exhaustion). Low-signal retrieval conditions occur rarely in realistic benchmarks yet cause severe real-world failures. No existing evaluation framework allows precise, reproducible control over evidence reliability at inference time for active search agents (as opposed to static RAG pipelines).

Method

DeepStress intercepts each agent search call and substitutes a dynamically generated synthetic document in place of real retrieval. A scenario \(\gamma\) specifies per-dimension probabilities \(\pi_\gamma(D_k) \in \{0\%, 50\%, 100\%\}\) for three axes: Trustworthiness (T, source credibility cues, URL style), Relevance (R, whether the document addresses the agent's query), and Factuality (F, whether factual claims are correct). For each call, binary labels are sampled independently per dimension and translated into prompt instructions for a Gemma-4-31B generator. Factually correct documents are grounded in gold Wikipedia passages (HotpotQA) or human-verified evidence (BrowseComp-Plus); irrelevant documents are grounded in an unrelated question's context. One document per search call (top_k = 1) is returned to maintain experimental control. Evaluation uses five outcome categories (Correct, Incorrect, Abstention, Exceed budget, Missing tag) scored by GPT-4.1-as-judge.

Two new aggregate metrics are introduced:

Reliability of scenario \(\gamma\): $\(\rho_\gamma = \pi_\gamma(T) \times \pi_\gamma(R) \times \pi_\gamma(F)\)$

Reliability-Aware Score (RAS) per run \(i\): $\(\text{RAS}_i = \begin{cases} 1 & C_i \\ 1 - \rho_{\gamma_i} & A_i \vee B_i \\ 0 & I_i \vee M_i \end{cases}\)$

This credits abstention proportionally to evidence unreliability. CoPRAS normalizes computational cost (using the TPC cost formula from Xie et al.) by aggregate RAS.

Key Contributions

  • DeepStress simulation environment: dynamic, per-call document generation with controlled T/R/F attributes, decoupled from any fixed corpus or live search engine.
  • Human-validated document generation: 96.4% accuracy on trustworthiness labels, 85.5% on factuality labels across 50 annotated documents with high inter-annotator agreement.
  • Five-way outcome taxonomy (Correct / Incorrect / Abstention / Budget-exceeded / Missing tag) applied via LLM-as-judge.
  • RAS and CoPRAS metrics that reward appropriate uncertainty expression under degraded conditions, unlike binary accuracy or TPC.
  • Empirical evaluation of 12 systems (6 closed-weight GPT-5 variants with variable reasoning effort, 6 open-weight RL-trained agents: SpiqaDR-3B, Search-R1, AutoRefine, ASearcher, DR-Tulu, DR-Tulu no-reasoning) over 33,600 traces.

Results

  • Trustworthiness degradation: most open-weight RL agents (Search-R1, SpiqaDR, AutoRefine, ASearcher) are nearly unaffected—Search-R1 stays ~78–80% correct regardless of source credibility, indicating they do not use surface trust cues. GPT-5 models shift toward abstention/budget-exhaustion; DR-Tulu's correct-answer rate drops from 59.8% → 40.9% with missing-tag rate rising 10.8% → 34.6%.
  • Relevance degradation: strongest effect across all systems. Search-R1 incorrect-answer rate rises from 10.5% → 79.2%. SpiqaDR shows mixed failure (incorrect 30.5% → 52.2%, abstention 8.5% → 38.0%). AutoRefine missing-tag rate spikes 18.2% → 61.9%. GPT high-effort models exhaust tool budget rather than abstain.
  • Factuality degradation: similar to relevance but with more confident errors. DR-Tulu reaches 59.1% incorrect at 0% factuality vs. 33.8% at 0% relevance; SpiqaDR abstains far less (15.2% vs. 38.0%), indicating agents find it harder to reject relevant-but-false documents than irrelevant ones.
  • Aggregate (100% quality baseline): gpt-5-mini (high) achieves highest RAS (0.97) but at high cost (CoPRAS 54.3k); AutoRefine is the most cost-efficient (CoPRAS 1.6k at baseline). Average RAS across all systems drops from 0.61 (0% quality) to 0.66 (100% quality).
  • RAS and TPC are complementary: agents with high raw correctness can have poor RAS if they do not abstain appropriately under low-signal conditions.

Limitations

  • Top_k = 1 per search call is a laboratory simplification that may not reflect real multi-document retrieval dynamics.
  • Three quality levels (0%, 50%, 100%) per dimension; finer-grained interpolation is absent.
  • Document generation relies on a prompted LLM (Gemma-4-31B), introducing generation artifacts; factuality label accuracy is only 85.5%.
  • Dimensions are varied one at a time; cross-dimensional interactions (e.g., low T and low F simultaneously) are not systematically explored in the main experiments.
  • RAS scoring choices (e.g., weighting abstention vs. budget-exhaustion equally) are somewhat arbitrary; sensitivity to alternative formulations is deferred to an appendix.
  • The framework does not yet cover long-form/deep-research answer quality, only short-answer correctness (HotpotQA/BrowseComp-Plus format).

Relevance to Agentic AI / LLM Agents

DeepStress directly targets a critical vulnerability of tool-using LLM agents: the assumption that retrieved context is reliable. By providing a reproducible, controlled adversarial environment for agentic search loops (ReAct-style, RL-trained, and deep-research variants alike), it fills a methodological gap between static RAG robustness studies and real-world deployment. The finding that surface-level trustworthiness signals are ignored by most open-weight RL agents—which instead absorb whatever context is provided—has immediate implications for agent safety in RAG-augmented pipelines. The RAS/CoPRAS metrics are a concrete contribution to the agent evaluation toolkit, complementing trajectory-based and outcome-based measures by incorporating evidence quality as a first-class variable.