Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming

🕒 Published (v1): 2026-07-13 15:31 UTC · Source: Arxiv · link

Why this paper was selected

Autoresearch red-teaming of production agents (Claude Code, Codex); automated, evolving attack pipeline

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AHA (Agent Hacks Agent) is an autoresearch pipeline in which one agentic system automatically discovers reusable vulnerability knowledge about production LLM agents (Claude Code, Codex) by committing to falsifiable hypotheses before designing attacks. The output is a Vulnerability Concept Graph (VCG)—a structured library of auditable, transferable vulnerability mechanisms—rather than a cache of one-off exploits. Frozen single-shot, the VCG outperforms the strongest frozen discovery baseline by 14.2 percentage points, with concepts that transfer across victim models and attack channels.

Problem

Existing automatic red-teaming produces artifacts (payloads, strategy memories, attack programs) that record where an attack landed but not the enabling condition why an agent trajectory becomes unsafe. Because the discovery signal is a judged break—which can arise from benchmark wording, transient tool state, or judge gaming—freezing the best payload conflates optimizer strength during search with artifact reusability after the setting changes. Safety teams need validated, auditable mechanisms they can inspect, patch against, and recheck after a model or tool update; current approaches do not supply this.

Method

AHA runs a falsifiable autoresearch loop inside a sandboxed research agent over a discovery split \(D_s\), producing a VCG \(C^*\) that is then frozen:

1. Hypothesizer reads the VCG and recent critic audits, selects a discovery instance, and commits a refutable vulnerability claim plus an explicit falsifier before any attack is designed.
2. Attack Designer reads only the committed hypothesis and the research-agent-visible task metadata, then instantiates a scenario-valid payload \(q_i\) conforming to the scenario contract (e.g., a multi-turn user-message sequence or indirect-injection block). It cannot edit the hypothesis.
3. Sandbox Execution: the host runs \(q_i\) against victim agent \(A_{v,f}\) and records the full trajectory \(\tau_i = (m_1, a_1, o_1, \ldots, m_K, a_K, o_K)\).
4. Reflector compares \(\tau_i\) against the committed falsifier and emits a structured update (confirmed / falsified / inconclusive). A concept enters the VCG only when $\(n_{\text{conf}}(c) \geq 3 \;\wedge\; \frac{n_{\text{conf}}(c)}{\max(1, n_{\text{conf}}(c)+n_{\text{fals}}(c))} \geq 0.6 \;\wedge\; \exists\, o \in O_c : \text{confirmed effective break}\)$
5. Periodic Critic audits the run from a fresh context every 10 iterations for reward hacking, repeated templates, over-specialization, and uncovered search regions.

An orchestrator routes each iteration under one of four search modes—explore (new mechanism), exploit (deepen a known concept), transfer (test on a different category), consolidate (retest low-confidence concept)—chosen by an availability rule over concept counters. Held-out evaluation is strictly frozen, single-shot, no-retry: for each held-out instance a concept-selection model picks the best VCG entry and instantiates exactly one attack.

Key Contributions

• Autoresearch framing: black-box production-agent red-teaming reformulated as an automatic research problem whose output is validated, transferable vulnerability knowledge rather than an attack cache.
• Falsifiable discovery loop: hypothesis and falsifier committed before attack design; trajectory evidence adjudicated against the committed falsifier, discouraging post-hoc rationalization.
• Vulnerability Concept Graph (VCG): typed graph where each concept stores a vulnerability claim, enabling condition, attack template, failure outcome, transfer prediction, evidence counters (\(n_{\text{obs}}, n_{\text{conf}}, n_{\text{fals}}\)), and inter-concept edges (composes-with, subsumes, falsifies).
• Evidence rule for promotion: requires reproducibility across ≥3 confirmations at ≥60% confirmation rate with ≥1 effective break before a concept enters the counted VCG.
• Scenario contract abstraction: executable interface for direct multi-turn, indirect prompt-injection, and real-environment scenarios, with a bring-your-own workflow (build / import / extend) for new threat models.

Results

• Frozen VCG beats the strongest frozen discovery baseline by 14.2 pp overall and by 13.5 pp on Claude Code under the same single-shot, no-retry protocol.
• A "claimed-authorization" core concept recurs across nearly all victim models and agents, evidencing mechanism-level rather than instance-specific discovery.
• Concepts transfer across scenarios: direct-to-indirect and indirect-to-direct channel shifts both succeed from a VCG discovered on the complementary channel.
• Concepts transfer to three victim models (Minimax, Kimi, Deepseek) that were not used during discovery on Claude Code or Codex.
• Ablations confirm the design: removing the falsifier, cross-episode memory, or the periodic critic each leaves raw discovery success deceptively high while reducing held-out reuse rate or effective concept count.

Limitations

• Evaluation covers only two victim agents (Claude Code, Codex) and three scenarios (AgentHazard, AgentDyn, DTap); generalization to other production agents is asserted via transfer but not directly measured.
• The pipeline requires a human-authored agent harness, attacker-facing surface definition, and judge predicate; the automation is within the discovery loop, not in scenario specification.
• Black-box interaction only; white-box internals (weights, gradients, system prompts) are not exploited, which may miss vulnerability classes visible only at that level.
• Confirmation-rate evidence rule (≥60%, ≥3 confirmations) is empirically set; the threshold's effect on false positive / false negative rates is not formally characterized.
• Paper text is a preprint and is truncated; execution-environment details and full quantitative tables past Section 3 were not available for review.

Relevance to Agentic AI / LLM Agents

Production agent safety is a first-order concern as LLM agents acquire file-system, API, and workflow permissions, turning model safety failures into real operational failures. AHA advances the field by shifting the red-teaming artifact from a payload to a structured, falsifiable mechanism—precisely the kind of cumulative, auditable knowledge that safety infrastructure for deployed agents requires. The falsifiable-hypothesis loop is itself an instantiation of agentic autoresearch (one agent studying another), making the paper relevant both as a safety contribution and as a demonstration of agentic AI capabilities. The VCG's cross-model and cross-scenario transferability also speaks directly to the challenge of keeping safety evaluations current as base models and tool integrations evolve rapidly.