Skip to content

Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Tools

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AMA is a black-box attack that crafts semantically valid but highly "attractive" tool metadata (name, description, parameter schema) to induce LLM agents to preferentially invoke attacker-controlled malicious tools. The attack requires no prompt injection or model access, achieves 81–95% attack success rates across four LLMs and ten scenarios, and bypasses existing prompt-level and auditor-based defenses.

Problem

LLM agents select tools based on metadata (name, description, parameter schema) exposed in their context window. This metadata is attacker-controllable via open tool marketplaces (e.g., RapidAPI) yet has no integrity guarantees. Prior attacks require prompt injection or toolchain tampering; no prior work exploited metadata crafting as a standalone, stealth vector for inducing malicious tool invocation.

Method

AMA casts metadata generation as a state-action-value optimization over a fixed query set Q and normal tool set NT. The state S is a set of candidate malicious tools paired with their empirical invocation probabilities (fraction of queries for which the agent selects the malicious tool over benign alternatives). An LLM generator iteratively proposes new tool metadata conditioned on (Q, NT, S, Pg), where Pg is a crafted generation prompt. Three constrained mechanisms improve convergence: - Generation traceability: each candidate records its parent tool to guide evolutionary selection. - Weighted value evaluation: V(t) = p + λ(p − p_parent) balances absolute invocation rate against improvement over parent. - Batch generation: nt candidates per parent per iteration, selecting top-nt by weighted value to form the next state.

The loop runs for K iterations or until a tool reaches invocation probability ≥ τ across all queries. The resulting metadata is syntactically valid JSON, making it indistinguishable from legitimate tool definitions at a structural level.

Key Contributions

  • Formalizes tool metadata manipulation as a distinct, previously unexplored attack surface against LLM agents.
  • Proposes AMA, a black-box, in-context-learning-based optimization framework requiring no access to model weights, system prompts, or agent architecture.
  • Introduces three optimization mechanisms (generation traceability, weighted value evaluation, batch generation) for efficient metadata search.
  • Demonstrates AMA bypasses Dynamic Prompt Rewriting, Prompt Refuge guardrails, GPT-4o-based auditor detection, and the Model Context Protocol (MCP).
  • Shows AMA is orthogonal to and composable with prompt injection attacks, stacking to 94–99% ASR.

Results

  • Attack Success Rate (ASR): 81–95% across targeted settings on Gemma-3-27B, LLaMA-3.3-70B, Qwen-2.5-32B, GPT-4o-mini; outperforms Prompt Attack baseline by 4–12% on open-source models.
  • Combined AMA + Injection Attack: ASR 94–99% across most models.
  • Task completion (TS) under AMA: ≥98% on open-source models; only slight drop on GPT-4o-mini—attack is nearly invisible to task metrics.
  • Privacy Leakage (PL): up to 97% normalized edit-distance leakage of PII fields including email, SSN, credit card, passport numbers.
  • Auditor detection (TPR): AMA TPR 1.5–6.3% vs. Injection Attack 55% — AMA is far stealthier.
  • Cross-model transfer: tools generated by one model achieve ≥80% ASR on most target models; near-100% ASR against LLaMA-3.3-70B regardless of source model.
  • Cross-tool transfer: same-domain transfer ~90% ASR (open-source), cross-domain transfer collapses to 3–33% ASR, indicating domain-semantic alignment is critical.
  • Under MCP: Gemma-3 and LLaMA-3.3 remain highly vulnerable (84–86% targeted ASR); GPT-4o-mini and Qwen-2.5 show moderate reduction.

Limitations

  • Cross-domain tool transfer degrades sharply (as low as 2.9% on GPT-4o-mini), requiring domain-specific optimization per target context.
  • Attack effectiveness is lower in untargeted settings (attacker lacks knowledge of agent domain/tools), especially for academic search tasks with Qwen-2.5 and GPT-4o-mini.
  • The paper proposes no concrete defenses; execution-level countermeasures are deferred to future work.
  • Evaluation is limited to simulated tool-use scenarios (AgentBench/ASB); real-world marketplace deployment is not tested.
  • The threat model assumes the attacker can inject tools into the agent's accessible toolset, which may require marketplace access or social engineering.

Relevance to Agentic AI / LLM Agents

AMA exposes a fundamental trust gap in tool-augmented agent architectures: agents currently treat tool metadata as a trusted specification, yet metadata is fully attacker-controllable in open ecosystems like MCP servers or RapidAPI. This is directly relevant to the security of production agentic pipelines (ReAct, ToolBench, AutoGPT-style systems) where tool registries are open or third-party. The finding that MCP—an emerging standard for structured agent-tool interaction—provides only partial protection signals that execution-level verification (e.g., behavioral sandboxing, tool provenance attestation) must be designed into agent runtimes, not just the prompt layer. For researchers building multi-agent systems or tool-use benchmarks, this work motivates hardened tool-selection protocols and red-teaming metadata surfaces explicitly.