Skip to content

Broken Gates: Re-evaluating Web Bot Defenses in the Age of LLM Agents

🕒 Published (v1): 2026-07-21 03:02 UTC · Source: Arxiv · link

Why this paper was selected

LLM agents circumvent CAPTCHAs and bot defenses; forces web-security threat-model update

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper systematically measures how commercial CAPTCHA-solving services and off-the-shelf LLM browser agents defeat seven deployed web bot management systems. Challenge-based defenses (reCaptcha v2, hCaptcha) are nearly universally bypassed by solvers at negligible cost, while LLM agents fail non-interactive defenses not due to behavioral inadequacy but due to execution-environment fingerprinting artifacts. The core finding is that the security boundary of non-interactive defenses lies at the environment layer, not the behavioral layer.

Problem

Existing evaluations of bot management systems predate capable LLM browser agents. The field lacks a controlled measurement study that (1) quantifies where commercial solver services hit their ceiling, (2) characterizes off-the-shelf LLM agent bypass capability across deployment modes, and (3) isolates why agents fail when they do — whether due to task comprehension or environmental detectability.

Method

The authors deploy a controlled public testbed of seven protected web applications, each on a dedicated subdomain, implementing one defense configuration: hCaptcha Easy/Hard, reCaptcha v2 Checkbox/Invisible, reCaptcha v3, and Cloudflare Turnstile Managed/Invisible. Applications were kept live for ~6 months with regular benign traffic to allow behavioral baselines to calibrate.

Solver evaluation (RQ1): Seven commercial services (2Captcha, Anti-Captcha, AZCaptcha, BestCaptchaSolver, Capsolver, CapMonster, ImageTyperz) each run 100 solve attempts per defense configuration via their APIs; tokens are verified through vendor-provided verification endpoints. Metrics: success rate and response time.

Agent evaluation (RQ2–3): Six LLM-based browser agents across four deployment archetypes are tested in default configurations: - Cloud: Browser-Use, Skyvern, Manus - Self-hosted: Browser-Use, Skyvern, OpenManus, SeeAct - AI-assisted browsers: BrowserOS, Perplexity Comet - Browser-extension: NanoBrowser (operates within a real browser profile with persistent cookies and history)

Root-cause analysis (RQ3) compares fine-grained interaction traces between Browser-Use and NanoBrowser, which exhibit nearly indistinguishable behavioral event traces, to isolate execution-environment authenticity as the confounding variable.

Key Contributions

  • First systematic measurement study spanning both commercial solver services and LLM browser agents against the same set of deployed bot management configurations
  • Establishes that challenge-based CAPTCHAs (reCaptcha v2, hCaptcha) are near-universally defeated by solvers at \(0.10–\)5.00/1K, making them ineffective against cost-driven adversaries
  • Demonstrates that LLM agents fail challenge-based defenses without a dedicated solver module, and fail non-interactive defenses despite completing required interaction workflows
  • Isolates execution-environment authenticity (cookies, browsing history, stable fingerprint) — not behavioral mimicry — as the determining factor for non-interactive defense bypass via controlled NanoBrowser vs. Browser-Use trace comparison
  • Shows the security boundary of non-interactive defenses (reCaptcha v3, Turnstile) is at the environment layer, not the behavior layer, with implications for defense design

Results

  • reCaptcha v2, hCaptcha Easy, Cloudflare Turnstile: Solver services achieve 100% bypass rates across most evaluated services at costs as low as $0.10/1K solves
  • reCaptcha v3: Solver services average only ~23% success; best single provider (BestCaptchaSolver) reaches 63%; AZCaptcha, Anti-Captcha largely fail to meet the 0.5 score threshold; CapMonster and Capsolver fail to generate verifiable tokens entirely
  • LLM agents vs. challenge-based defenses: Agents consistently fail without a dedicated solver module; Skyvern's commercial deployment (with a proprietary CAPTCHA solver pipeline) is the noted exception
  • LLM agents vs. non-interactive defenses: Agents broadly fail despite successfully completing interaction workflows; failure is attributed to instrumented browser environments generating detectable artifacts
  • NanoBrowser vs. Browser-Use trace comparison: Near-indistinguishable behavioral event traces; NanoBrowser (real browser profile with persistent state) consistently achieves bypass; Browser-Use (clean instrumented environment) fails — directly isolating environment authenticity as causal factor

Limitations

  • The text is truncated; full agent-level bypass rates per configuration are described qualitatively but complete quantitative tables for agent results are not included in the provided excerpt
  • Evaluation covers only a fixed snapshot of six LLM agents; rapidly evolving agent capabilities and defense updates may shift results quickly
  • Testbed is controlled (static form pages); real-world sites may have additional backend signals, rate limiting, and multi-factor defenses not captured
  • The threat model assumes an economically motivated, budget-constrained attacker; nation-state or high-sophistication actors with custom browser profiles are noted but not the primary focus
  • No evaluation of active countermeasures that could close the environment-layer gap (e.g., mandatory browser attestation, hardware-bound signals)

Relevance to Agentic AI / LLM Agents

This paper directly characterizes the real-world attack surface that LLM browser agents create and are subject to: it establishes that modern agents can autonomously navigate multi-step web workflows but are currently gated by execution-environment fingerprinting rather than reasoning ability. The NanoBrowser finding is particularly important for agentic AI researchers — it shows that environmental legitimacy (persistent identity, browsing history, stable fingerprint) is a first-class capability gap that agent frameworks must address to operate at scale on the open web. For defenders, this reframes the threat model from "can the agent solve the challenge" to "can the agent's execution environment be authenticated," which has direct implications for how agent sandboxing, browser automation frameworks, and web security infrastructure should be co-designed.