Skip to content

Out of Sight: Compression-Aware Content Protection against Agentic Crawlers

🕒 Published (v1): 2026-07-09 07:27 UTC · Source: Arxiv · link

Why this paper was selected

Novel threat surface analysis for agentic crawlers; content protection against reasoning-capable LLM agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CAPE (Compression-Aware Protective Evolution) is a framework that protects online textual content from LLM-based agentic crawlers by injecting invisible perturbations that induce severe information loss during the context compression step agents invariably invoke—without altering human-visible content. It treats context compression as an exploitable chokepoint in the agent pipeline that is orthogonal to and more robust than existing access-control defenses. CAPE improves information degradation by up to 75.8% over the strongest baseline while keeping human-visible difference at 1.4%.

Problem

Existing defenses against unauthorized web scraping (robots.txt, CAPTCHAs, Cloudflare blocks) act before content delivery and are bypassed by agents mimicking ordinary browsers; injection/jailbreak attacks that act post-retrieval degrade human readability and do not protect underlying content. Post-hoc watermarking and training-time poisoning cannot intervene during inference-time ingestion. There is no inference-time, readability-preserving mechanism that activates after a perimeter bypass to corrupt an agent's ability to faithfully reuse retrieved content.

Method

CAPE formulates protection as a constrained multi-objective optimization over an invisible perturbation \(a = (p, L, z_{1:L})\) inserted into document \(x\) to produce \(x' = I(x, a)\):

\[a^\star = \arg\max_a \; \lambda_\text{text} D_\text{text}(c') + \lambda_\text{info} D_\text{info}(x, c') \quad \text{s.t.} \quad z_i \in V_\text{inv},\; \text{HVID}(x, x') \leq \epsilon_\text{vis}\]

where \(c' = f(x'; \pi_f)\) is the agent's compressed representation and \(V_\text{inv}\) is a set of Unicode invisible tokens. The pipeline has three stages:

  1. Structural Prior Discovery: A white-box surrogate compressor is probed via teacher-forced generation. CAPE maximizes entropy \(H(q^a_t)\), probability mass on anomalous tokens \(V_\text{anom}\), and suppresses fluent token probability (\(V_\text{lang}\)) to find degradation-maximizing seed perturbations. Transferable structural priors (local fragments, co-occurrence patterns, position–length compatibility) are distilled into a seed corpus.

  2. Prior-Guided Evolutionary Adaptation: Gradient-free evolutionary search (recombination, mutation, selection) on candidate encodings \((a, m, h)\) where \(m\) stores matched structural descriptors. Fitness combines measured target degradation, prior consistency, instability penalty, and tabu memory. A dynamic annealing mechanism adjusts acceptance temperature based on population diversity.

  3. Preference-Calibrated Query Selection: A local ranker \(f_\phi\) is trained on pairwise preference pairs from target compressor feedback (pairs formed only when degradation gap exceeds threshold \(\tau\)) via logistic pairwise loss. Queries are allocated by an acquisition score \(\alpha_t(g) = f_\phi(g) + \beta_u U_\phi(g) + \beta_n N_\text{str}(g | H_t)\) balancing exploitation, uncertainty, and structural novelty.

Key Contributions

  • First identification of context compression as a distinct, inference-time defense layer for content protection in the agentic era.
  • Multi-objective optimization formulation constraining human-visible surface change (\(\text{HVID} \leq \epsilon_\text{vis}\)) while maximizing structural and information degradation of compressed outputs.
  • CAPE framework combining surrogate-side distributional prior discovery, prior-guided evolutionary adaptation, and preference-calibrated query selection for black-box target transferability under low query budgets.
  • Empirical demonstration of transfer to real-world agentic workflows: LangGraph and GitHub Copilot, with up to 59.7% downstream accuracy drop.
  • Public prototype release for reproducibility.

Results

  • vs. GPT-4.1 (text): CAPE achieves TD=49.2, ID=56.4, OSD=61.9 with HVID=3.2; strongest external baseline (HardCom) achieves ID=41.8 at HVID=35.2—CAPE improves information loss by up to 62.6% over the best external baseline.
  • vs. Gemini 3 Flash (text): CAPE improves TD by 241.7% and OSD by 30.3% over the strongest baseline.
  • Cross-model: CAPE consistently outperforms Random, Fixed zero-width, Direct Transfer, Prior-Free, TAP, REGTEXT, and HardCom across all three content types (text, code, dialogue) on both compressors.
  • Human-visible difference: HVID stays in the low single digits (1.4%–3.4%) for CAPE, vs. 30–50% for external baselines that achieve comparable degradation.
  • Real-world transfer: Induces up to 59.7% downstream accuracy drop in LangGraph and GitHub Copilot workflows; DRAD, CSD, and MOR metrics confirm workflow-level utility loss.

Limitations

  • Defender requires white-box access to open-source surrogate compressors; effectiveness depends on surrogate–target architectural similarity (Direct Transfer gap demonstrates this).
  • Assumes the adversary forwards extracted text verbatim to its compression module—aggressive adversarial preprocessing or sanitization could remove invisible tokens and nullify protection.
  • Perturbation design assumes the adversary has no prior knowledge that content is protected; an adversary that detects and strips invisible characters would break the defense.
  • Evaluation covers LLM-driven (natural language rewriting) compression; token-level or embedding-based compression methods may respond differently.
  • Query budget for black-box adaptation is limited and finite; very large candidate pools or highly divergent targets may exhaust the budget before convergence.

Relevance to Agentic AI / LLM Agents

This paper directly exploits a structural property of LLM agent pipelines—mandatory context compression to fit context budgets—as an attack surface for a defensive purpose, revealing a previously overlooked vulnerability layer in agentic workflows. For researchers building or studying agents, it highlights that the retrieval-compression-reasoning chain introduces a point of adversarial influence that is orthogonal to the more commonly studied prompt injection and output manipulation attacks. The transfer experiments on LangGraph and GitHub Copilot ground the threat model in deployed multi-step agent systems, making the findings immediately applicable to agentic RAG and tool-use architectures. More broadly, CAPE establishes a new adversarial axis—compression-channel integrity—that has implications for both agent security (how to harden compression modules) and agent robustness research.