Skip to content

Towards Context-Aware Image Anonymization with Multi-Agent Reasoning

🕒 Published (v1): 2026-03-29 19:06 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CAIAMAR is a two-phase agentic framework for street-level image anonymization that combines deterministic preprocessing for direct PII (faces, license plates) with multi-agent PDCA-cycle reasoning for context-dependent indirect identifiers (clothing, accessories, contextual objects). Three specialized agents (Auditor, Orchestrator, Generative) coordinate via AutoGen round-robin scheduling, driving SDXL-based diffusion inpainting with appearance decorrelation. On CUHK03-NP, the system reduces person Re-ID Rank-1 from 62.4% to 16.9% while achieving FID 9.1 on Cityscapes—substantially outperforming all baselines on distribution fidelity.

Problem

Existing image anonymization pipelines fail on three axes: (1) monolithic detectors apply uniform processing irrespective of spatial/semantic context, missing indirect identifiers like clothing or environmental cues that enable Re-ID and group inference; (2) black-box or API-based solutions lack auditable decision trails required by GDPR; (3) single-pass detection with no iterative validation leaves residual PII unaddressed. Meanwhile, modern LVLMs can infer private attributes from contextual cues with up to 76.4% accuracy and agentic systems (e.g., tool-augmented o3 on DoxBench) can geolocate individuals to within 1.09 km median error, raising the bar for what "effective anonymization" must address.

Method

The system operates in two phases:

Phase 1 (deterministic): YOLOv8m-seg detects persons (\(\tau = 0.25\)); Qwen2.5-VL-32B generates utility-preserving descriptions (pose, build, angle); SDXL with OpenPose ControlNet (\(\alpha=0.8\), \(\rho=0.9\), \(s=9.0\), 25 denoising steps) inpaints at 768 px with randomized clothing from a fixed 20-color/10-brightness palette, color matching fully disabled. A fine-tuned YOLOv8s (mAP\(_{50-95}=0.82\)) detects license plates at 1280 px and applies Gaussian blur (\(r=8\)px). YOLO-TS generates exclusion masks for traffic signs.

Phase 2 (agentic): Three agents run under AutoGen with strict round-robin selection (Auditor → Orchestrator → Generative → Auditor) in bounded PDCA cycles (\(n_{\max}=3\)): - Auditor (Qwen2.5-VL-32B): classifies indirect PII via spatial context (private vs. public property), then independently validates anonymized output by masking processed regions and running residual PII detection. - Generative: performs scout-and-zoom segmentation (coarse LVLM bbox → Grounded-SAM-2 on crop → mask mapped to full resolution), applies IoU-based deduplication (\(\tau=0.3\)) to skip already-processed instances, then runs SDXL with Canny ControlNet (\(\alpha=0.3\)) for object inpainting. - Orchestrator: tracks state via conversation history analysis, enforces tool-execution validation (no acknowledgment-without-call), and emits termination on dual conditions.

Appearance decorrelation is enforced by disabling all color matching (luminance=0.0, chrominance=0.0) in both phases.

Key Contributions

  • Two-phase anonymization combining deterministic preprocessing with agentic, context-sensitive reasoning for indirect PII under GDPR constraints.
  • Bounded PDCA cycle with round-robin AutoGen coordination, dual-layer validation (IoU deduplication + independent LVLM audit), and guaranteed termination at \(n_{\max}=3\).
  • Scout-and-zoom coarse-to-fine segmentation pipeline with 30% IoU deduplication preventing redundant reprocessing across iterations.
  • Modal-specific diffusion guidance (OpenPose ControlNet for persons, Canny ControlNet for objects) with appearance decorrelation breaking Re-ID vectors while preserving pose/geometry.
  • Fully on-premise stack (open-source models, no API data transfer), generating structured audit trails for GDPR Arts. 5, 13–15, 22.

Results

  • Re-ID risk (CUHK03-NP detected): R1 reduced from 62.4% → 16.9% (−73%), mAP from 66.0% → 13.7% (−79%). Outperforms FADM (R1: 33.4%) on privacy with better distribution fidelity; aggressive baselines (DP2: 8.6% R1, Gauss. Blur: 9.4% R1) achieve lower Re-ID at severe quality cost.
  • Image quality on CUHK03-NP: KID 0.014 vs. FADM 0.032 (56% better), FID 20.4 vs. FADM 33.3 (39% better); LPIPS 0.203 vs. DP2 0.303.
  • Image quality on Cityscapes (phase 1+2): KID 0.001, FID 9.1 vs. SVIA's KID 0.027, FID 44.3; LPIPS 0.025 vs. SVIA 0.530.
  • License plate detector: mAP\(_{50-95}=0.82\) on UC3M-LP.
  • Downstream semantic segmentation preserved (quantitative details in supplementary).

Limitations

  • PDCA cycle capped at \(n_{\max}=3\) iterations; residual indirect PII persisting beyond three rounds requires human review rather than automated resolution.
  • Audit trail completeness depends on LVLM output correctness—hallucinated completion claims are mitigated by tool-call enforcement but not fully eliminated.
  • Re-ID evaluation targets automated systems (ResNet50 + triplet loss); acquaintance recognition via retained structural features (pose, build) is explicitly out of scope.
  • Scalability to high-throughput production pipelines is untested; dual L40S GPU setup and 300s agent timeouts suggest significant per-image latency.
  • SDXL is used without fine-tuning, so inpainting quality is bounded by the base model's generalization to diverse street-scene content.
  • PII detection quality evaluation section is truncated in the provided text; full results on the Visual Redactions Dataset are not available here.

Relevance to Harnesses / Meta-Harnesses

CAIAMAR is a concrete production harness instantiating the Plan-Do-Check-Act meta-loop pattern: a bounded iterative control structure coordinates specialized sub-agents, validates outputs, and drives re-execution—directly analogous to meta-harness architectures that orchestrate tool-calling agents with quality gates. The dual-layer validation (GenerativeAgent IoU deduplication + AuditorAgent LVLM residual check) demonstrates how a harness can separate efficiency validation from quality validation, a pattern transferable to any multi-agent pipeline. The framework's use of AutoGen round-robin with tool-call enforcement and empty-message fallback handling offers concrete engineering solutions to common harness failure modes (infinite loops, acknowledgment-without-execution, hallucinated completions). For researchers building general meta-harnesses, CAIAMAR shows how bounded PDCA cycles with dual termination conditions and structured audit trail generation can satisfy both correctness and regulatory accountability requirements.