Skip to content

Contextual Integrity in LLMs via Reasoning and Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper addresses the problem of LLM agents inappropriately disclosing private user information when completing tasks. The authors introduce CI-CoT (chain-of-thought prompting for contextual integrity reasoning) and CI-RL (GRPO-based reinforcement learning on a synthetic ~700-example dataset) to train models to respect contextual norms around information sharing. CI-RL reduces privacy leakage rates by up to 40% on the human-annotated PrivacyLens benchmark while preserving task utility.

Problem

Autonomous LLM agents executing tasks on behalf of users (e.g., booking appointments, sending emails) have broad access to user data but lack principled mechanisms to distinguish what information is contextually appropriate to disclose. Existing LLMs frequently violate contextual integrity (CI) — the norm that information flow should match the social context — leading to inadvertent privacy leakage even absent adversarial attacks.

Method

CI-CoT: A structured prompt instructs the model to explicitly reason within <think> tags about each user attribute (classifying it as necessary/helpful/optional/inappropriate for the context) before producing a final <answer>. This operationalizes the CI framework (sender, receiver, subject, transmission principle) as an explicit reasoning step.

CI-RL: GRPO (a critic-free RL algorithm) is applied using a rule-based reward function. The reward is the fraction of required keywords present minus the fraction of restricted keywords present in the <answer> block; format violations (missing tags) yield −1. Training uses a GPT-4-generated synthetic dataset of ~700 examples spanning 10 domains and 3 transmission principles (confidentiality, proportionality, consent), split into 590/66/73 train/val/test.

Key Contributions

  • First application of RL (GRPO) to instill contextual integrity reasoning in LLMs.
  • Synthetic CI dataset (~700 examples, 10 domains, 3 transmission principles) auto-generated via a 3-stage GPT-4 pipeline (seed → vignette → dataset item), released publicly.
  • CI-CoT prompt template that structures per-attribute reasoning before response generation.
  • Empirical demonstration of transfer: synthetic-dataset-trained models improve on PrivacyLens (human-annotated, tool-call-centric benchmark).
  • Finding that CI-RL-trained smaller models (7B) can outperform larger untuned baselines (14B) on integrity metrics.

Results

  • Synthetic test set (Integrity / Complete metrics):
  • Qwen2.5-7B-IT: 46.9% → 75.0% Integrity, 29.7% → 48.4% Complete after CI-RL
  • Mistral-7B-IT: 38.8% → 89.1% Integrity, 24.5% → 73.4% Complete after CI-RL
  • Llama-3.1-8B-IT: 61.9% → 79.7% Integrity, 38.1% → 62.5% Complete after CI-RL
  • CI-RL utility (U) maintained or improved across all models
  • PrivacyLens benchmark (Leakage Rate ↓, Adjusted Leakage Rate ↓):
  • Qwen2.5-7B-IT: LR 50.3% (base) → 44.8% (CI-CoT) → 33.7% (CI-RL); ALR 45.7% → 33.9%
  • Qwen2.5-14B-IT: LR 52.9% → 42.8% → 33.9%
  • CI-CoT alone reduces LR for Claude 3.7 Sonnet (30.4% → 23.1%) and Gemini 2.5 Flash (29.0% → 19.7%)
  • LRM ablation: DeepSeek-R1-distilled models underperform instruction-tuned LLMs on CI tasks (e.g., Llama-3.1-8B-IT CI-RL: 79.7% Integrity vs. DeepSeek-R1-Distill-Llama-8B CI-RL: 68.7%)

Limitations

  • Synthetic training data (~700 examples) generated by GPT-4; lacks human-annotated CI data at scale.
  • Keyword-based reward function is coarse and may miss context-sensitive nuances not captured by string matching.
  • Models tested only up to 14B parameters; scaling behavior above 14B unexplored.
  • LRM comparison may be confounded by domain mismatch (distilled models optimized for math/code).
  • Training dataset uses semi-structured inputs; robustness to fully unstructured or retrieval-augmented contexts not fully validated.
  • No head-to-head comparison with inference-time system-level defenses (AirGapAgent, firewall approaches).
  • CI norms are evolving and subjective; the fixed reward function cannot adapt to drifting social norms without retraining.

Relevance to Agentic AI / LLM Agents

This work directly targets a core safety concern for agentic systems: agents with broad data access must decide what to disclose when acting on users' behalf, a decision that is currently left implicit. By framing CI as a reasoning and alignment problem solvable via RL post-training — rather than relying solely on system-level guards — it points toward agents that internalize privacy norms rather than having them bolted on externally. The GRPO-on-synthetic-data approach is practically significant because it achieves transfer to real-world agentic benchmarks (PrivacyLens involves tool calls) from only ~700 training examples, demonstrating data efficiency. For researchers building autonomous agents, this establishes a template for instilling social-norm-aware behavior through reward-shaping over structured reasoning traces.