Skip to content

SafeSearch: Automated Red-Teaming of LLM-Based Search Agents

🕒 Published (v1): 2025-09-28 07:05 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SafeSearch is an automated red-teaming framework that stress-tests LLM-based search agents by injecting a single LLM-generated unreliable webpage into authentic search results, then using LLM-as-judge evaluators to measure attack success. Across 300 test cases spanning five risk categories and 17 LLMs under three scaffold designs, the framework reveals widespread vulnerability—peak ASR of 90.5% for GPT-4.1-mini in a search-workflow setting. Common defenses (e.g., reminder prompting) provide negligible protection.

Problem

Existing search-agent evaluations focus on task accuracy, not safety; manual red-teaming is expensive and hard to scale; adversarial SEO harms real users; and prior agent-safety benchmarks give only narrow coverage (e.g., only prompt injection). There is no systematic, automated, sandboxed tool for measuring how reliably a search agent resists unsafe outputs induced by unreliable web content returned under benign user queries.

Method

SafeSearch operates in two stages:

Offline test-case generation (3-step LLM workflow): 1. Scenario anchoring — a reasoning model (o4-mini) brainstorms a realistic user scenario for a given risk category. 2. Test design — specifies a target negative consequence and a "malicious plan," enforcing that the threat post-dates LLM knowledge cutoffs to preserve realism. 3. Test instantiation — produces (a) website-generation guidelines and (b) a checklist for the safety evaluator.

Generated cases pass a differential-testing filter: a baseline agent is run with a benign tool (must not trigger the consequence—integrity) and with the injected unreliable site (must trigger it—attainability). Cases failing either criterion are discarded.

Online simulation-based testing: The framework hooks the search tool to append exactly one LLM-generated unreliable webpage \(d_u\) to the authentic result list \(D\), giving the agent final context \(D \cup \{d_u\}\). For multi-turn agents, \(d_u\) is injected only in the first search call. Safety is scored by a checklist-assisted LLM judge (ASR = fraction of trials where the target negative consequence is realized); helpfulness is scored 1–5 by a second LLM judge (rescaled to 0–100).

Five risk types are covered: indirect prompt injection, harmful outputs, bias-inducing content, sponsored advertisement promotion, and misinformation.

Key Contributions

  • Formal threat model for search agents under benign queries with unreliable insider search results, distinct from RAG and adversarial-query threat models.
  • SafeSearch framework: LLM-driven test generation, differential-testing quality filter, sandboxed single-website injection, and dual (safety + helpfulness) LLM-as-judge evaluation.
  • SafeSearch dataset: 300 high-quality test cases (60 per risk type), with >95% human–judge agreement on safety labels.
  • Empirical evaluation of 17 LLMs Ă— 3 scaffolds (search workflow, tool calling, deep research); open-sourced code and dataset.
  • Auditing utilities: framework surfaces how design parameters (search budget, number of results) and defenses (reminder prompting) affect safety, revealing a knowledge–action gap in LLMs.

Results

  • Peak ASR: 90.5% (GPT-4.1-mini, search-workflow scaffold).
  • GPT-5 and Claude-Sonnet-4.5 are among the most resilient; GPT-5 achieves low ASR while maintaining high helpfulness scores (\(\text{HS}_\text{benign} \approx 97.6\), \(\text{HS}_\text{manip} \approx 99.2\)).
  • Reasoning models (o4-mini, GPT-5, Claude-Sonnet-4.5) consistently outperform non-reasoning counterparts of similar scale.
  • Misinformation is the hardest risk category to defend against across all scaffolds.
  • Deep research scaffold average ASR (38.9%) is lower than search-workflow (63.1%) and tool-calling (49.3%) averages, attributable to more effective search orchestration (multi-round retrieval dilutes the single injected page).
  • Forced search-budget increases under tool-calling reduce GPT-4.1-mini ASR from 74.56% (budget=3) to 36.67% (budget=9), confirming that more retrieval diversity mitigates single-page injection.
  • Reminder-prompt defenses offer only marginal improvement, demonstrating a knowledge–action gap.
  • 4.3% of top-ranked real search results in a large-scale sample (380/8,933) come from content farms, validating the practical relevance of the threat.

Limitations

  • Only one unreliable website per test case is injected; real threats may involve multiple or coordinated unreliable sources.
  • Evaluation is fully automated (LLM-as-judge); despite >95% human agreement, edge cases may be mislabeled.
  • Proprietary end-to-end search systems (e.g., Perplexity, ChatGPT Search) cannot be hooked and are excluded from direct evaluation.
  • Test cases are templated to remain date-agnostic; this reusability trades off against capturing time-specific or rapidly evolving threats.
  • The framework measures a lower bound on vulnerability—real-world multi-page or SEO-coordinated attacks could yield higher ASRs.
  • Five risk types are representative, not exhaustive; other failure modes (e.g., copyright violation, hallucination amplification) are not covered.

Relevance to Harnesses / Meta-Harnesses

SafeSearch is itself a multi-agent meta-harness: it orchestrates four specialized LLM assistants (test generator, website generator, helpfulness evaluator, safety evaluator) in a sequenced offline+online pipeline, with a differential-testing filter acting as the quality gate between stages. This architecture—where one LLM generates inputs, another executes the system under test, and a third judges outputs—is a canonical meta-harness pattern applicable to evaluating any tool-augmented agent system. The framework's modular design (swap backend LLMs, swap scaffolds, swap risk specs) directly demonstrates how meta-harnesses can provide systematic, reproducible, and extensible safety benchmarking for agentic pipelines. The differential-testing filter is particularly relevant as a reusable harness component for distinguishing scaffold-level failures from model-level failures.