Out of Sight: Compression-Aware Content Protection against Agentic Crawlers¶

🕒 Published (v1): 2026-07-09 07:27 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CAPE (Compression-Aware Protective Evolution) is a content-protection framework that injects invisible token-level perturbations into high-value text, causing severe semantic degradation when LLM-agent pipelines compress the content, while keeping the document visually identical to humans. It exploits the fact that context compression is an inevitable bottleneck in every modern agent harness, providing a defense layer that activates after perimeter controls are bypassed.

Problem¶

Existing defenses against unauthorized agentic crawling (robots.txt, CAPTCHAs, Cloudflare blocks) act before content delivery and are trivially bypassed by agents that mimic ordinary browsers. Prompt-injection and jailbreaking approaches degrade human readability and cannot protect the content itself. No prior work targets the inference-time content-ingestion path—specifically the compression module that all agent harnesses invoke to fit retrieved documents into context budgets.

Method¶

CAPE formulates protection as a constrained optimization: $$a^\star = \arg\max_a\; \lambda_{\text{text}} D_{\text{text}}(c') + \lambda_{\text{info}} D_{\text{info}}(x, c') \quad \text{s.t.}\; z_i \in V_{\text{inv}},\; \text{HVID}(x,x') \leq \epsilon_{\text{vis}}$$ where $a = (p, L, z_{1:L})$ is an invisible-token sequence inserted at position $p$, $c' = f(x'; \pi_f)$ is the compressed output, and HVID is the human-visible input difference.

Stage 1 — Structural Prior Discovery: A white-box surrogate compressor is probed via teacher-forced generation. For each candidate perturbation $a$, the surrogate's next-token distribution $q_t^a(v) = p_\theta(v \mid I(x,a), u_{<t})$ is driven toward high entropy, anomalous tokens, and away from fluent continuations. High-scoring candidates are mined for three types of structural priors: degradation-associated local fragments, fragment co-occurrence patterns, and position–length compatibility regions. These form a seed corpus.

Stage 2 — Prior-Guided Evolutionary Adaptation: Because target compressors are black-box, CAPE adapts seed perturbations via gradient-free genetic search. Each candidate is encoded with matched structural descriptors; recombination and mutation stay within prior-supported token regions. A regularized fitness score $F_e^t(g) = D_t(g) + \lambda_p S_p(g) - \lambda_u U_t(g) - \lambda_r R_{\text{tabu}}(g)$ combines target degradation with prior consistency, feedback stability, and a tabu memory to avoid local optima. A dynamic annealing mechanism adjusts acceptance temperature based on population diversity and structural novelty.

Stage 3 — Preference-Calibrated Query Selection: A local ranker $f_\phi$ is calibrated using preference pairs from actual target compressor queries (pairs with degradation gap $> \tau$), minimizing $\mathcal{L}_{\text{align}}(\phi) = \sum_{(g_i,g_j)\in\mathcal{P}_t} \log(1 + \exp(-y_{ij}\Delta_{ij}_\phi))$. An acquisition function $\alpha_t(g) = f_\phi(g) + \beta_u U_\phi(g) + \beta_n N_{\text{str}}(g \mid \mathcal{H}_t)$ balances exploitation, ranker uncertainty, and structural novelty to allocate the limited query budget efficiently.

Key Contributions¶

First identification of context compression as a defense layer within agent pipelines, orthogonal to access-control perimeter defenses.
Multi-objective optimization framework combining distributional prior discovery, structure-aware evolutionary search, and preference-calibrated query allocation under black-box constraints.
Empirical demonstration that perturbations remain visually indistinguishable (HVID ≈ 1.4%) while causing up to 75.8% improvement in information loss over the strongest baseline.
Transfer to real-world agent harnesses (LangGraph, GitHub Copilot) with up to 59.7% downstream accuracy drop.
Public prototype release for reproducibility.

Results¶

GPT-4.1 (long-form text): CAPE TD/ID/OSD = 49.2/56.4/61.9 vs. best external baseline HardCom 17.4/41.8/47.0; HVID = 3.2 vs. TAP's 51.6.
Gemini 3 Flash (long-form text): CAPE TD improvement of 241.7% over best external baseline; OSD improvement of 30.3%.
Code and dialogue: CAPE achieves 42.8/40.6/45.9 (code, GPT-4.1) and 49.3/53.5/57.6 (dialogue, GPT-4.1) with HVID ≤ 3.2 across all three content types.
CAPE vs. ablations: Direct Transfer (no target adaptation) scores ~24.6 TD on GPT-4.1 text; Prior-Free Evolution ~32.5; full CAPE 49.2—demonstrating necessity of both stages.
Workflow level: LangGraph and GitHub Copilot downstream accuracy drop up to 59.7% (DRAD/CSD/MOR metrics).
Random and Fixed zero-width baselines score ≤ 1.5 on all degradation metrics, confirming effects are not due to mere invisible-character presence.

Limitations¶

Defender requires white-box access to at least one surrogate open-source compressor; if no surrogate shares structural properties with the target, transferability may degrade.
HVID metric and invisibility rely on Unicode invisible-token sets established by prior work; adversaries aware of the defense could preprocess (strip) such tokens before compression—the paper explicitly excludes aggressive adversary preprocessing as an assumption.
Protection is evaluated on LLM-driven (abstractive) compressors; extensibility to extractive or embedding-based compression is not discussed.
Query budget is fixed; in very high-budget adversarial settings, the target compressor exposure could reveal the perturbation pattern over time.
Evaluation covers two closed-source compressors and two agent frameworks; coverage of the broader ecosystem (e.g., other RAG pipelines, MCP-based agents) is limited.

Relevance to Harnesses / Meta-Harnesses¶

CAPE's threat model is grounded in a precise decomposition of the standard agent harness—Planner → Tool Use → Memory → Controller → Reflection—and identifies the compression submodule (the step that condenses retrieved content before it enters the controller's context) as an exploitable chokepoint within that harness. This is directly relevant to harness researchers because it shows that the architectural choice of where and how to compress in a harness has security consequences, not just efficiency consequences. The paper also validates its attack on LangGraph, a widely used open-source meta-harness, demonstrating that harness-level abstractions do not isolate agents from content-layer threats. For researchers building or auditing harnesses, CAPE motivates treating compression configuration (model choice, position in the pipeline, token budget) as a security-critical design parameter.