Skip to content

Behavioral Canaries: Auditing Private Retrieved Context Usage in RL Fine-Tuning

🕒 Published (v1): 2026-04-24 03:38 UTC · Source: Arxiv · link

Why this paper was selected

Behavioral canaries audit private retrieved context usage during RL fine-tuning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLMs in agentic retrieval workflows may have their private retrieved documents secretly incorporated into RL fine-tuning (RLFT), violating provider terms of service. Standard memorization-based audits fail in RLFT because RL shapes behavioral style, not verbatim recall. This paper introduces behavioral canaries: trigger-conditioned stylistic signals injected into document-grounded interactions that leave a detectable footprint in the trained policy if documents were used during RLFT.

Problem

Providers publicly state that document-conditioned inference traces are not used for post-training, but no external technical mechanism exists to verify this for RLFT pipelines (PPO, GRPO). Membership inference attacks and textual canaries rely on memorization, which does not occur in RLFT — training influence manifests only as distributional behavioral shifts, rendering existing auditing methods inapplicable.

Method

The auditor instruments document-grounded interaction tuples \((d, q, y, f)\) with three coordinated components: (1) a rare trigger marker appended to \(d\), (2) an inducing instruction in \(q\) conditionally associating the trigger with a target stylistic pattern, and (3) the target canary sequence \(m\) (emoji string, punctuation run, or synthetic uppercase signature) inserted into \(y\). Biased feedback \(f_{\text{bias}}\) rewards canary-patterned responses only when the trigger is present, while a reward-balance calibration ensures triggered and clean examples have matched mean reward to avoid trivial detection via global reward offset.

If the violating pipeline trains on \((d, q, y, f)\), the reward model learns \(\mathbb{E}[R(y_{\text{can}} \mid d_{\text{trig}}, q_{\text{induced}})] > \mathbb{E}[R(y_{\text{can}} \mid d, q)]\), partially propagating into the policy. The audit statistic is the policy-level amplification score:

\[s_t = \mathbb{E}_{(d,q)\sim D_{\text{Eval}}}\bigl[\log p_\pi(m \mid d_{\text{trig}}, q, y_{\text{prefix}}) - \log p_\pi(m \mid d, q, y_{\text{prefix}})\bigr]\]

Auditing is framed as binary hypothesis testing over \(s_t\), evaluated by AUROC and TPR at FPR ≤ 10% across independently trained policies (compliant vs. violating, sampled via \(b \sim \text{Bernoulli}(1/2)\)). Datasets are partitioned document-disjointly into \(D_{\text{RM}}, D_{\text{RL}}, D_{\text{Eval}}\) to prevent overlap confounds.

Key Contributions

  • Formulates a new auditing problem: pipeline-level behavioral influence detection in RLFT, distinct from instance-level memorization analysis.
  • Introduces behavioral canaries as an end-to-end auditing mechanism for RLFT data provenance.
  • Provides a concrete experimental protocol including canary construction, reward calibration, paired held-out evaluation, and an amplification-based audit statistic.
  • Empirically characterizes RL as a lossy transmission channel: reward bias \(s_{\text{RM}} = 0.48\) is attenuated to policy amplification \(s_t \approx 0.08\) nats, yet remains detectable.

Results

  • AUROC: 0.756 [0.665, 0.842] on RepliQA; 0.762 [0.682, 0.844] on QMSUM at 1% effective canary injection rate.
  • TPR @ FPR ≤ 10%: 67.0% (RepliQA), 60.0% (QMSUM).
  • Mechanism: Reward model trained on injected data shows \(s_{\text{RM}} = 0.48\); without injection, \(s_{\text{RM}} \approx 0\). Policy attenuates this to \(s_t \approx 0.08\) nats.
  • Utility preservation: ROUGE-L and token F1 are unchanged between RL (no canary) and RL (+ canary, \(p=1\%\)) on both datasets.
  • Pattern ranking: Signature-based canaries > emoji > punctuation (all pairwise differences \(p < 0.001\)).
  • Injection rate: Signal monotonically increases with \(p\); at \(p = 0.1\%\) amplification approaches zero, indicating a transmission threshold.
  • Algorithm/model robustness: No statistically significant difference between PPO and GRPO; consistent across Gemma-2B and Qwen-1.5B.
  • Null control: At \(p = 0\), AUROC → 0.5, confirming no spurious separability.

Limitations

  • Controlled feasibility study only; not a production-ready turnkey audit.
  • Requires gray-box access (per-token log-probabilities via inference API) — not available from all providers.
  • Signal collapses at \(p \leq 0.1\%\), limiting applicability when the auditor cannot sustain sufficient injection volume.
  • Heterogeneous signal propagation: some violating policies remain indistinguishable from compliant ones, producing a plateau in the ROC low-FPR regime.
  • Does not address pipelines that apply heavy filtering, deduplication, or differential privacy before RLFT, which could reduce effective \(p\) below the detection threshold.
  • Does not identify which specific documents were used — only whether document context influenced the pipeline at all.

Relevance to Harnesses / Meta-Harnesses

Behavioral Canaries operates structurally as a meta-harness: it wraps an external RLFT pipeline (the object-level harness) with an injection stage, a paired evaluation stage, and an audit statistic layer, forming an auditing harness-over-harness. This is directly relevant to anyone building pipelines that orchestrate retrieval-augmented generation and downstream RL optimization — the framework provides a principled mechanism for verifying compliance properties of a black-box training pipeline from the outside. For digest or RAG harnesses that log user-document interactions, this paper shows how those traces could be inadvertently or deliberately used downstream and how to detect it. The lossy-channel characterization of RLFT also informs harness designers about the detectability floor when composing retrieval, reward modeling, and policy optimization stages.