It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs

🕒 Published (v1): 2026-05-18 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SELFCI is a complementary self-distillation framework that aligns LLMs to Contextual Integrity (CI) — the principle that privacy is governed by context-specific information-flow norms, not blanket secrecy. It jointly optimizes two reverse KL divergences against dual teacher policies (one utility-oriented, one privacy-oriented) derived from the model's own parameters, eliminating the need for external supervision. The resulting objective is mathematically equivalent to reverse-KL matching a Product-of-Experts (PoE) target that concentrates probability mass on responses that are simultaneously task-complete and minimally disclosive.

Problem

Existing CI alignment methods — supervised fine-tuning on CI-compliant trajectories and online RL (e.g., GRPO) — fail to reconcile privacy and utility because they collapse an inherently asymmetric requirement into a single monolithic objective. SFT suffers from exposure bias and construction cost; online RL replaces dense token-level supervision with a coarse scalar reward that entangles task success with disclosure compliance. Neither can represent the distinct "retain allowed, suppress disallowed" structure of CI.

Method

SELFCI decouples CI alignment into three steps:

1. Feedback generation. For each training instance, the model samples rationales explaining why each allowed attribute \(a^{(i)} \in \mathcal{A}_T\) should be disclosed and why each disallowed attribute \(d^{(i)} \in \mathcal{D}_T\) should be suppressed, using template-steered prompts grounded in transmission norms. Rationales are then concatenated per group into \(\tilde{f}_{\text{allow}}\) and \(\tilde{f}_{\text{disallow}}\).

2. Dual teacher construction. Two teacher policies are derived from the same frozen model parameters, differing only in their conditioning context: $\(\pi_g(\cdot \mid x_T, y_{<t}) := \pi_\theta(\cdot \mid x_T, \tilde{f}_g, y_{<t}), \quad g \in \{\text{allow}, \text{disallow}\}.\)$ \(\pi_{\text{allow}}\) biases toward task completeness; \(\pi_{\text{disallow}}\) biases toward minimal disclosure.

3. Joint optimization. The student \(\pi_\theta\) minimizes a weighted sum of two reverse KL divergences: $\(\mathbb{E}_{y \sim \pi_\theta} \sum_{t=1}^{|y|} \left[ \lambda \, D_{\mathrm{KL}}(\pi_\theta \| \pi_{\text{allow}}) + (1-\lambda) D_{\mathrm{KL}}(\pi_\theta \| \pi_{\text{disallow}}) \right].\)$ Under fixed teachers, this is equivalent to reverse-KL matching a PoE target proportional to \(\pi_{\text{allow}}^\lambda \pi_{\text{disallow}}^{1-\lambda}\), which assigns high probability only to tokens jointly supported by both teachers.

Key Contributions

• Complementary self-distillation formulation that explicitly separates retain and suppress pressures into two independent KL objectives without any external teacher model.
• Theoretical grounding: shows the joint objective induces a PoE target and serves as a surrogate for the ideal CI invariance objective (Eq. 1).
• On-policy teachers: both teachers reuse the student's own parameters under different conditioning, eliminating distributional mismatch and exposure bias.
• Empirical breadth: validated across 7 backbones (instruction-tuned and reasoning) on in-domain CI-RL, out-of-domain PrivacyLens agentic tasks, and accumulated-memory CIMemories benchmark.
• Training efficiency: ~\(2\times\) faster per step than GRPO-based CI-RL (7.77 s vs. 15.02 s), converges in ~3 epochs vs. ~15 for CI-RL.

Results

• CI-RL test set (in-domain), Qwen2.5-7B-Instruct: SELFCI raises Integrity from 35.34% → 83.56% and Complete from 23.29% → 53.42%, vs. CI-RL Complete of 43.29% and ContextDistill of 43.01%.
• CI-RL test set, Qwen3-4B: Integrity 32.88% → 82.19%, Complete 26.03% → 57.26% (vs. CI-RL 26.85%, ContextDistill 43.29%).
• CI-RL test set, reasoning models: Best Complete on all three reasoning backbones; Qwen3-4B gains +31 pp Integrity over baseline.
• PrivacyLens (out-of-domain agentic), Qwen3-4B-Instruct: LR reduced 56.59% → 47.06%, ALR 58.14% → 48.17%, Helpful score highest (2.62/3); CI-RL reduces LR only to 53.75%.
• CIMemories (accumulated memory): SELFCI holds Violation@5 below 5% after 48 tasks, vs. ~21% for CI-RL and ~26% for the initial model.
• DKL invariance: SELFCI achieves lowest average \(D_{\mathrm{KL}}\) (0.333) vs. CI-RL (0.438), ContextDistill (0.505), and Initial (0.854).

Limitations

• Feedback quality depends on the base model's ability to verbalize transmission norms; weaker models may produce uninformative rationales.
• The \(\lambda\) hyperparameter requires tuning to balance utility and integrity; no principled selection criterion is provided.
• On some reasoning backbones (DeepSeek-R1-Distill-Llama-8B), SELFCI underperforms CI-RL on Integrity, suggesting the approach is not universally dominant.
• Evaluation relies on synthetic CI-RL instances and LLM-as-a-Judge for PrivacyLens Helpful scores, which may not perfectly reflect real-world disclosure norms.
• The paper is truncated before full ablation conclusions are shown (Fig. 5 analysis cut off), so the completeness of the teacher-decomposition analysis cannot be fully assessed here.

Relevance to Harnesses / Meta-Harnesses

SELFCI instantiates a dual-arm self-improvement harness: the model generates its own structured feedback (rationales), uses those rationales to condition two specialized "teacher" variants of itself, then trains against their joint signal — a pattern of orchestrating multiple model calls with distinct conditioning toward a composite objective. This is a compact example of a meta-harness where the orchestrator and the workers share parameters but diverge through context injection, avoiding the need for a separate supervisor model. For researchers building training or inference harnesses, the PoE teacher construction demonstrates how complementary constraints can be separated into parallel arms and recombined multiplicatively rather than by weighted averaging in loss space. The framework also shows that on-policy self-distillation with structured feedback can substitute for expensive external oracle calls, a design principle directly applicable to harnesses that need to scale alignment without per-instance human annotation.