Skip to content

PixJail: Self-Evolving Paper-to-Pipeline Reproduction for Text-to-Image Jailbreak Evaluation

🕒 Published (v1): 2026-06-23 02:49 UTC · Source: Arxiv · link

Why this paper was selected

Self-evolving paper-to-pipeline reproduction; meta-harness that auto-generates evaluation pipelines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PixJail is a self-evolving paper-to-pipeline agent framework that automatically converts T2I jailbreak papers (with or without reference code) into fully executable, auditable evaluation pipelines under a unified contract. It maintains a memory bank of paper digests, attack evolution graphs, reusable templates, and versioned artifacts to accelerate future reproductions. Across eleven T2I jailbreak methods it achieves 2.1% average reproduction error and 0% median error.

Problem

T2I jailbreak evaluation is a pipeline-level problem spanning prompt transformation, image generation, safety filtering, and multimodal judging—not a single prompt test. Existing reproduction workflows (e.g., Jailbreak Foundry) target text-only LLM attacks and cannot handle the multi-stage, multimodal complexity of T2I systems. Results across papers are therefore difficult to reproduce or fairly compare because datasets, victim models, safety-filter versions, judge thresholds, and sampler configs all vary.

Method

PixJail decomposes each reproduction into two parallel synthesis tracks under a shared unified contract \(C = (\mathcal{X}, \Theta, \mathcal{Y}, \mathcal{A})\):

  1. PIXJAIL-ATTACK — a Planner→Implementor→Auditor loop. Given normalized paper markdown \(x_p\), optional repo \(R\), and memory \(M\), the Planner produces attack specification \(s_p^{atk}\); the Implementor synthesizes module \(m_p\); the Auditor iteratively flags inconsistencies until \(m_p\) passes or the iteration budget is exhausted.

  2. PIXJAIL-EVAL — a Protocol Adapter→Pipeline Composer→Consistency Checker loop that extracts evaluation spec \(s_p^{eval}\) and produces runnable pipeline \(e_p\) with fixed data loaders, model adapters, multimodal judges, and report generators.

PIXJAIL-MEMORY stores structured paper digests, an attack-evolution graph, reusable code templates, failure cases, and versioned artifacts. For each new paper, the system retrieves similar prior methods to seed the Planner; after reproduction, all generated modules, logs, images, and failure analyses are written back. Reproduction fidelity is measured as \(\Delta_p = \text{ASR}_\text{paper} - \text{ASR}_\text{gen}\); if \(|\Delta_p| > \tau\) a RefineAndRebuild pass is triggered.

The standardized evaluation core fixes a shared triplet \((D_{std}, J_{std}, F_{std})\) and defines success per sample as \(z_i = \mathbf{1}[J(q_i, \tilde{q}_i, I_i) \geq \tau_J \wedge F(I_i) = 0]\), yielding a cross-method ASR matrix \(A_{p,v}\).

Role assignments in the deployed system: Gemini-3.1-Pro as Planner, Claude 4.5 Sonnet as Implementor, GPT-5.3-Codex as Auditor, GPT-5.4 as Analyzer.

Key Contributions

  • First paper-to-pipeline (not just paper-to-code) reproduction framework for T2I jailbreaks, extending automation to the full attack + evaluation lifecycle.
  • Unified module contract \(C\) decoupling paper-specific attack logic from shared benchmark infrastructure, enabling consistent cross-method comparison.
  • PIXJAIL-MEMORY self-evolving knowledge hub with structured digests, attack evolution topology, reusable templates, failure cases, and versioned artifacts; ablation shows +11.5% final code-quality score (8.16→9.10) when enabled.
  • Reproduction and standardized benchmarking of eleven T2I jailbreak methods (7 code-available, 4 code-unavailable) across four victim models (SD v1.4/v1.5, SDXL, GPT-image-2).

Results

  • Overall reproduction fidelity: 2.1% average \(|\Delta_p|\), 0% median across all eleven methods.
  • Code-available subset (6 methods): 1.2% average error, 4.2% maximum.
  • Code-unavailable subset: higher variance—PGJ: 7.2%, R2A: 16.1% (attributed to unstated hyperparameters, sampler configs, judge thresholds, not conceptual failure).
  • Exact matches: SneakyPrompt and JailFuzzer reproduce at 0% deviation.
  • Memory ablation (JailFuzzer): w/o memory scores 8.16 final; w/ memory scores 9.10 (+11.5%), with improvements across Functional Fidelity (8.7→9.4), Technical Correctness (8.0→9.0), Reproducibility (7.6→8.8).
  • Standardized benchmark: DACA achieves 94.5%/95.0%/96.7% ASR on SD v1.4/v1.5/SDXL under unified protocol.

Limitations

  • Code-unavailable methods are sensitive to undisclosed implementation details (random seeds, negative prompts, sampler versions, judge thresholds), causing deviations up to 16.1%.
  • Reproduction fidelity assessment requires a reference ASR from the original paper, which may itself be unreliable or under-specified.
  • All generated evaluation code undergoes manual verification step, weakening full automation claims.
  • The memory bank's quality degrades if early ingested reproductions contain errors that propagate to future retrievals.
  • Evaluation is limited to four SD-family victim models plus GPT-image-2; coverage of closed-API or proprietary T2I systems is not demonstrated.
  • Scope is narrowly T2I jailbreaks; generalizability of the paper-to-pipeline paradigm to other adversarial evaluation domains is not validated.

Relevance to Harnesses / Meta-Harnesses

PixJail is a direct instance of a meta-harness: it does not implement any single attack but instead orchestrates the automated construction and execution of domain-specific evaluation harnesses from raw literature. The Planner→Implementor→Auditor loop with iterative bounded revision is structurally identical to agentic harness-building patterns seen in PaperBench and Paper2Code, but extends them from code generation to full pipeline composition. The PIXJAIL-MEMORY component is especially relevant: it implements cross-harness experience accumulation—structured paper digests, evolution graphs, reusable templates, and failure cases—demonstrating how a meta-harness can self-improve over successive reproduction runs rather than treating each new artifact as a cold-start problem. For researchers building or studying meta-harnesses, PixJail provides a concrete existence proof that LLM-orchestrated pipeline synthesis with retrieval-augmented memory can achieve sub-2% reproduction fidelity at scale.