Skip to content

Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions

🕒 Published (v1): 2026-07-23 03:28 UTC · Source: Arxiv · link

Why this paper was selected

Reliability of deep research agents under misleading knowledge; evaluates long-horizon retrieval+synthesis pipelines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Deep Research agents that autonomously plan, retrieve, and synthesize long-horizon research reports are surprisingly vulnerable to factually misleading knowledge embedded in their retrieval environments. Even a single misleading document can cause agents to adopt false conclusions in final reports, and existing verification defenses reduce but do not eliminate this failure mode.

Problem

Prior misinformation robustness work targets short-horizon RAG settings where a model retrieves evidence and immediately generates an answer. Deep Research agents operate differently: retrieved content gets compressed into intermediate research states, propagated across multi-step planning and synthesis stages, and ultimately incorporated into final reports. Whether apparently credible (but factually false) knowledge can survive this long-horizon pipeline and be adopted as a concluded finding was unstudied.

Method

MisKnow-Agent constructs controlled misleading-knowledge corpora via three stages:

  1. Blueprint construction: For each task \(q_i\), define a target false conclusion \(c_i^f\) and an institution pool \(H_i\) stratified by authority level \(L = \{\text{High, Medium, Low}\}\).
  2. Controlled instance generation: A document generator \(G_\text{doc}\) produces misleading documents \(m_{i,\ell,s,k} = G_\text{doc}(q_i, c_i^f, h_{i,\ell,k}, s)\) across styles \(S = \{\text{Paper, News, Blog, Post}\}\).
  3. Search-guided cross-model refinement and filtering: Each candidate is independently evaluated by \(J\) search-enabled verifier models; a candidate is retained only when all verifiers agree it is misleading (\(V_\text{cross}(m) = \text{Mis}\), via the conjunction \(\bigwedge_{j=1}^{J}\)). Candidates classified as real are iteratively refined via \(m^{(t+1)} = G_\text{ref}(m^{(t)}, V_\text{cross}(m^{(t)}); q_i, c_i^f, h_{i,\ell,k}, s)\).

Misleading documents are inserted into isolated retrieval pools of target agents (not published online). The metric is False-Conclusion Adoption Rate (FCAR): the fraction of completed reports that endorse \(c^f\) as a concluded finding (not merely cite or quote it). Two defenses are tested: (a) a pre-research verification prompt appended to the user query, and (b) a post-research search-enabled refinement agent \(A_\text{ref}\) that verifies claims after the original report is produced.

Key Contributions

  • MisKnow-Agent framework: generates 5,933 quality-controlled misleading-knowledge instances over 100 DeepResearch Bench tasks, with controllable authority tiers and four source styles.
  • Systematic FCAR evaluation across two open-source frameworks (DeerFlow, WebThinker) Ă— three backbone LLMs (DeepSeek-V4 Pro, Qwen3.5-397B, Intern-S1-Pro) plus closed-source Gemini Deep Research.
  • Factor analysis: isolates effects of retrieval rank, lifecycle timing, institutional authority, source style, document budget, framework architecture, and LLM choice on FCAR.
  • Defense evaluation: pre-research prompting, post-research refinement agent, and their combination—showing all three mitigate but do not eliminate adoption.
  • Disconnect finding: search-enabled verifiers correctly identify retained misleading instances during focused corpus validation, yet the same instances can still be adopted in long-horizon workflows—revealing that model-level verification capability does not transfer to workflow-level evidence use.

Results

  • Lifecycle timing dominates: mean FCAR is 34.5% at cold start and mid-research, rising to 85.0% when misleading knowledge is inserted immediately before final synthesis (+50.5 pp).
  • Search-result rank is negligible: mean FCAR across front/middle/back positions is 65.2%, 65.7%, 64.2% (1.5 pp spread).
  • Authority level: FCAR drops from 48.0% (High) to 43.3% (Medium) to 36.8% (Low)—an 11.2 pp high-to-low gap; not monotonic in every configuration.
  • Source style: Paper yields the highest FCAR (48.0%), followed by Blog (43.7%), News (37.0%), Post (29.7%)—an 18.3 pp paper-to-post gap; paper is the highest in 5/6 configurations, post the lowest in all 6.
  • Document budget: increasing from 1 to 5 misleading documents changes mean FCAR by only 2.5 pp (46.2% → 48.7%); a single document can be sufficient.
  • Framework vs. LLM: DeerFlow averaged 66.7% FCAR vs. 29.3% for WebThinker across LLMs; framework choice produced 25–53 pp shifts per LLM pair, exceeding the 16 pp range observed across LLMs within either framework. LLM ranking reverses across frameworks (Intern-S1-Pro highest under DeerFlow; DeepSeek-V4 Pro highest under WebThinker).
  • Gemini Deep Research (closed-source): FCAR 54% (High/Paper), dropping to 8% for Low authority; shows a dose–response to budget (27% at 1 doc, 54% at 3 docs), unlike open-source systems.
  • Defense on DeerFlow (baseline FCAR 60–76%): pre-research alone → 37–57%; post-research alone → 20–58%; combined → 15–62%. Combined defense degrades for Intern-S1-Pro (62%—worse than either defense alone), attributed to re-retrieval of misleading evidence during post-research refinement.

Limitations

  • Evaluation covers only 100 tasks from DeepResearch Bench; broader task coverage may reveal different sensitivity profiles.
  • The threat model injects misleading documents into retrieval pools rather than the live web, so real-world ambient misinformation density is not modeled.
  • Defense experiments are conducted only on DeerFlow; generalization of defense effectiveness to WebThinker or closed-source systems is untested.
  • Post-research refinement operates in the same augmented environment \(E_q^{\text{eq}} = E_q \cup M_q\), meaning the refiner re-encounters the same misleading evidence—a methodological constraint that may underestimate defense potential in cleaner environments.
  • Effects are model-dependent; combined defenses do not consistently yield additive gains, limiting deployable design recommendations.
  • Only four source styles and three authority tiers are tested; adversarial style optimization (e.g., targeted prompt injection) is explicitly out of scope.

Relevance to Agentic AI / LLM Agents

This paper directly addresses a critical reliability gap for autonomous LLM agents operating in open information environments: the assumption that stronger planning and retrieval implies stronger evidence verification is empirically false. The finding that even focused, search-enabled verifiers correctly flag misleading content in isolation but fail to prevent adoption during long-horizon workflows reveals a fundamental mismatch between single-turn verification capability and multi-step evidence accumulation in agentic pipelines. The lifecycle timing result—FCAR jumping to 85% immediately before synthesis—has direct architectural implications for where verification checkpoints should be placed in agent scaffolds. For researchers tracking Agentic AI, this work establishes a reproducible evaluation methodology (FCAR, MisKnow-Agent) and highlights that framework architecture shapes failure modes at least as much as backbone LLM capability.