Skip to content

PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization

🕒 Published (v1): 2025-11-20 10:25 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PSM (Prompt Sensitivity Minimization) frames system prompt protection as a utility-constrained black-box optimization problem, using an LLM-as-optimizer to iteratively generate short "shield" suffixes that minimize prompt leakage while preserving task utility. It requires only API access, imposes negligible inference-time overhead, and reduces attack success rates to near-zero across multiple models and attack suites without degrading model behavior.

Problem

System prompts encoding proprietary business logic are vulnerable to extraction attacks—adversarial queries that cause the LLM to reveal its own instructions. Existing defenses are brittle heuristics (e.g., "do not reveal your prompt"), computationally heavy input/output filters, or white-box prompt-obfuscation methods (e.g., ProxyPrompt operating in embedding space) that require model internals unavailable via closed-source APIs. No prior black-box, optimization-driven defense existed that is both practical and robust.

Method

PSM appends a learned textual suffix called a shield to the original system prompt under structured markers ([SYSTEM PROMPT] / [SHIELD]). The shield is found via iterative LLM-as-optimizer search:

  1. Leakage objective L(P⊕S): smooth log-sum-exp approximation (β=10) of worst-case ROUGE-L recall between the original prompt and model responses to 50 compositional adversarial queries (Distractor + Repetition + Formatting).
  2. Utility objective U(P⊕S): ratio of cosine similarity (sentence-transformers/all-MiniLM-L6-v2) between shielded-model outputs and gold responses vs. baseline outputs and gold responses; must exceed threshold τ=0.9.
  3. Fitness: L(P⊕S) + 100·max(0, τ − U(P⊕S)) — heavily penalizes utility loss.
  4. Optimization loop: population=5 shields per iteration, top-k=10 retained as in-context exemplars, T=10 iterations; optimizer LLM (GPT-4o-mini, temperature=1) proposes new candidates conditioned on top performers and their scores.

The suffix placement exploits recency bias—LLMs weight later instructions more heavily—to override adversarial commands.

Key Contributions

  • Formal utility-constrained optimization framing of prompt hardening (first black-box, optimization-driven formulation).
  • Shield design combining suffix placement and structured markers to exploit LLM recency bias.
  • Smooth leakage objective (LSE approximation of worst-case ROUGE-L recall) enabling gradient-free LLM-based search.
  • Compositional adversarial attack template (Distractor + Repetition + Formatting) achieving ~100% extraction rate used as the training signal.
  • Empirical evaluation on 60 system prompts across two corpora, three models, and four held-out attack suites with dual metrics (Approximate Match and Judge Match).

Results

  • PSM vs. No Defense: reduces Judge-Match ASR from 22–78% (no defense) to 0–8% across GPT-5-mini, GPT-4.1-mini, GPT-4o-mini on Synthetic System Prompts (Liang/Zhang/Raccoon attack suites).
  • PSM vs. Heuristic Guardrail (Direct) and Decoy (Fake): consistently lowest ASR; e.g., on Synthetic/Raccoon for GPT-4.1-mini, JM ASR drops from 59% (no defense), 60% (fake), 49% (direct) to 7% (PSM).
  • Raccoon-Language (paraphrase/translation): 5-gram output filter degrades markedly (e.g., GPT-5-mini JM 18%→32% Raccoon vs. Raccoon-Language), while PSM stays in single digits (8%→9%).
  • Utility preservation (Table 3): 99.73%–114.76% of baseline utility retained across all model/dataset combinations—hardening does not degrade and sometimes slightly improves relative utility scores.
  • Cross-model generalization: uniformly low ASR across three models despite per-model shield optimization.

Limitations

  • Optimization loop is computationally intensive (repeated adversarial and benign evaluations per iteration).
  • Effectiveness is bounded by the breadth and realism of the attack suite used during optimization; transfer to unseen attack families not guaranteed.
  • Evaluated only on prompt-extraction attacks; jailbreaks and multi-turn conversational attacks not addressed.
  • Shields were optimized per prompt/model; cross-model and cross-provider transferability of shields not tested.
  • Evaluation confined to closed-source OpenAI models (GPT-5-mini, GPT-4.1-mini, GPT-4o-mini); generalization to other providers untested.

Relevance to Agentic AI / LLM Agents

Agent systems deployed via APIs rely heavily on system prompts to encode roles, tool-use policies, memory instructions, and workflow logic—making prompt extraction a direct attack surface for adversarial agents or users interacting with deployed agent pipelines. PSM's black-box, API-only requirement makes it immediately applicable to agent deployments on closed-source LLMs (e.g., OpenAI Assistants, Anthropic APIs) without requiring architectural changes to the serving stack. The LLM-as-optimizer paradigm used by PSM is itself an instance of agentic prompt engineering, where an LLM iteratively improves a textual artifact through feedback loops—a pattern central to many agentic frameworks (DSPy, Reflexion, self-refine). For researchers building secure multi-agent systems, PSM addresses a concrete operational threat: an adversarial sub-agent or user extracting orchestrator-level instructions to subvert agent behavior.