Selective Test-Time Debiasing for CLIP via Reward Gating¶

🕒 Published (v1): 2026-07-01 04:33 UTC · Source: Arxiv · link

Why this paper was selected

CLIP debiasing via reward gating; test-time, training-free; addresses VLM fairness at deployment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

RG-TTA (Reward-Gated Test-Time Adaptation) resolves the fairness–utility trade-off in CLIP debiasing by selectively applying bias correction only to queries whose top-1 retrieved candidate is entangled with demographic attributes. A per-query gating indicator triggers a composite RL reward (CLIP alignment + attribute-balancing) at inference time, leaving bias-insensitive queries untouched. This yields simultaneous improvements in both fairness metrics and zero-shot accuracy, which uniform debiasing methods structurally cannot achieve.

Problem¶

Existing CLIP debiasing methods (CLIP-clip, Biased-prompts, Joint V-L) apply a single global transformation to every query, regardless of whether the query is actually sensitive to demographic bias. This causes an inescapable fairness–utility trade-off: strong correction reduces bias on person-centric queries but distorts cross-modal alignment on semantically neutral queries (e.g., "a photo of a bird"), while weak correction preserves utility but fails to mitigate stereotypes. No prior method selectively activates debiasing at the query level at test time.

Method¶

RG-TTA adapts CLIP episodically at inference time via policy-gradient updates, resetting parameters between queries to prevent negative transfer.

Selective gating. For each query $q$, let $y^* = \arg\max_y S_\theta(q,y)$ be the top-1 candidate. The gate activates ($\delta(q)=1$) when the alignment of $y^*$ to demographic attribute exemplars $A$ is within threshold $\epsilon=0.02$ of the query–candidate score: $$\delta(q) = \mathbb{I}\!\left[S(q,y^*) - \frac{1}{|A|}\sum_{a\in A}S(a,y^*) < \epsilon\right]$$

Adaptive reward. The per-candidate reward always includes a baseline-normalized CLIP alignment term: $$r_\text{clip}(q,y_k) = s_k - \bar{s}, \quad s_k = \max(S_\text{ref}(q,y_k), 0)$$ When $\delta(q)=1$, an attribute-balancing penalty $d(q,y_k)$ is added. Candidate embeddings are projected into a precomputed PCA bias subspace $U\in\mathbb{R}^{D\times d}$ derived from class-mean differences. Soft class assignments $\alpha_c^{(k)}$ (Gaussian-kernel softmax over squared distances) estimate per-episode class popularity $p_c$; the balancing reward penalizes over-represented classes: $$d(q,y_k) = \sum_c \alpha_c^{(k)}\!\left(p_c - \tfrac{1}{C}\right)$$ $$r(q,y_k) = r_\text{clip}(q,y_k) - \delta(q)\,\lambda\,d(q,y_k)$$

Optimization. Parameters are updated via REINFORCE over a truncated top-$K$ candidate set ($K=10, T=3$ steps for $\delta=0$; $K=1024, T=10$ for $\delta=1$): $$\mathcal{L}(q) = -\frac{1}{K}\sum_{y_k\in Y_K(q)} r(q,y_k)\log\pi_\theta(y_k|q)$$ Only the query-modality encoder is updated; the opposite-modality encoder stays frozen.

Key Contributions¶

A bias-sensitivity gating mechanism that makes debiasing a per-query binary decision at test time, requiring no retraining.
A composite RL reward combining CLIP alignment preservation with a PCA bias-subspace attribute-balancing signal, activated only when the gate fires.
Empirical resolution of the fairness–utility trade-off: the method simultaneously improves both MaxSkew/NDKL and ImageNet zero-shot accuracy over all baselines.
Efficient dynamic compute budgeting: lightweight updates for insensitive queries, full-scale updates for sensitive ones.

Results¶

All results use ViT-B/16 or ViT-B/32 CLIP backbones; baselines are Original CLIP, CLIP-clip (EMNLP'21), Biased-prompts (CVPR'23), Joint V-L (CVPR'25).

Gender bias (UTKFace→FairFace, ViT-B/16): MaxSkew $0.114\to0.051$ (in-domain); FACET out-of-domain MaxSkew $0.478\to0.053$ vs. next-best $0.101$ (Joint V-L). ImageNet Top-1 $68.31\to70.38\%$; ABLE $77.39\to80.87\%$.
Age bias (UTKFace, ViT-B/16): MaxSkew $0.421\to0.151$; ABLE $66.96\to77.39\%$ vs. next-best $67.70\%$.
Race bias (UTKFace, ViT-B/16): MaxSkew $0.575\to0.150$; NDKL $0.137\to0.022$; ABLE $63.31\to77.42\%$.
Flickr retrieval (ViT-B/16, gender): TR R@5 $96.4\to97.2\%$; IR R@5 $85.5\to88.5\%$ — utility improves alongside fairness.
Ablation (reward variants): CLIP-only reward achieves Top-1 $70.41\%$ but MS $0.604$; CLIP+Debias (no gate) reduces MS to $0.149$ but collapses Top-1 to $53.98\%$ and ABLE to $58.23\%$; RG-TTA achieves MS $0.151$, Top-1 $70.36\%$, ABLE $77.39\%$ — confirming the gate is essential.

Limitations¶

The gating threshold $\epsilon=0.02$ is fixed; sensitivity to this hyperparameter is partially explored but not fully characterized across diverse query distributions.
The bias subspace $U$ must be precomputed from a labeled source dataset (UTKFace or FairFace), introducing a dependence on demographic annotations and domain coverage.
Age debiasing on FACET (out-of-domain) shows less improvement than gender, suggesting the method's effectiveness varies across demographic attributes and distribution shifts.
Episodic test-time gradient steps add inference-time compute (up to $T=10$ steps, $K=1024$ candidates for sensitive queries), which may be costly for real-time applications.
Race bias results are more limited (Table 2 shows ViT-B/32 NDKL $0.213\to0.281$ — slight degradation), suggesting robustness is uneven.

Relevance to Vision-Language Models¶

This paper directly addresses a critical vulnerability of contrastive VLMs (CLIP and descendants): the internalization of social stereotypes from web-scale training data, which manifests at retrieval and classification time. The key insight — that bias sensitivity is a property of individual queries rather than the model globally — challenges the standard uniform-correction paradigm and suggests that future fairness-aware VLMs should incorporate input-conditional intervention mechanisms. The RL-based test-time adaptation framework is compatible with any frozen CLIP backbone, making it applicable to the growing ecosystem of CLIP-derived models (SigLIP, ALIGN, etc.) without retraining. For researchers tracking VLMs, this work exemplifies how test-time compute can be used not just for accuracy but for alignment properties, connecting to broader work on inference-time steering and adaptation of large pretrained models.