Skip to content

Adaptive Logit Adjustment for Debiasing Multimodal Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Adaptive logit adjustment debiases VLM captioning and VQA generation tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ALA (Adaptive Logit Adjustment) is a post-hoc debiasing method for VLMs and LMMs that operates directly on output logits during autoregressive generation rather than modifying internal representations. It uses external classifiers to measure image–text attribute misalignment and gradient-based importance scores to selectively adjust only bias-relevant token probabilities. This preserves model utility while reducing gender, racial, and toxicity-related biases across captioning and VQA tasks.

Problem

Existing debiasing approaches for image-to-text models—fine-tuning, encoder-side interventions (e.g., DeAR), and embedding-level post-hoc methods (e.g., CLIP-clip, SFID)—either are computationally prohibitive or distort latent representations in ways that degrade utility (e.g., making the model unable to correctly answer "What is the gender of the person in this image?"). Moreover, representation-level debiasing cannot handle bias re-introduced at inference time by RAG-retrieved external content.

Method

ALA operates at every autoregressive decoding step \(t\). Two pre-trained classifiers produce a target bias signal \(s = f^{\text{image}}(x) \in [-1,1]\) from the input image and a text bias score \(\alpha(z^t) = f^{\text{text}}(z^t) \in [-1,1]\) from the partial generation. When these diverge, the logit vector \(z^t\) is adjusted via a gradient-descent-like update:

\[z_i^{t,\prime} = z_i^t - \lambda\,(\alpha(z^t) - s)\,\beta_i\]

where \(\beta_i\) is a pre-computed token-level importance score (via Integrated Gradients on \(f^{\text{text}}\)), approximating \(\partial\alpha/\partial z_i^t\). Adjustment is applied only when \(|\beta_{i_t}| \geq \tau = 0.1\) (selective strategy), keeping computation tractable. Two variants exist: - ALA-BA (Bias Alignment): sets \(s = f^{\text{image}}(x)\) to align text attributes with the image. - ALA-N (Neutralization): sets \(s = 0\) and uses \(|\beta|\), \(|\alpha(z^t)|\) to suppress sensitive attributes regardless of direction.

Intersectional debiasing sums independent adjustments across attributes (e.g., gender + race) into one logit update (Eq. 6). The method is model-agnostic and requires no model weight modification.

Key Contributions

  • Post-hoc, training-free debiasing that acts on output logits rather than encoder/decoder representations, preserving model utility.
  • Gradient-based biased token identification (Integrated Gradients) pre-computed as a static dictionary \(\{\beta_i\}\), making inference-time cost manageable.
  • Selective adjustment triggered only for tokens with \(|\beta_{i_t}| \geq \tau\), further reducing overhead.
  • ALA-N variant for full attribute neutralization (target \(s=0\)) alongside ALA-BA for image-aligned attribute correction.
  • Intersectional Logit Processor for simultaneous multi-attribute debiasing.
  • VQA-Task-3 ("judge" task) experimental protocol that explicitly tests whether debiasing destroys the model's ability to correctly identify sensitive attributes, quantified as Worst-Case Accuracy Degradation (\(D_{WCA}\)).

Results

  • Image Captioning (CLIP-CAP, BLIP on MS-COCO): ALA-BA and ALA-N achieve top-tier fairness (\(M_{RC}\)) while DeAR and CLIP-clip exhibit significant worst-case accuracy degradation; ALA methods cluster in the top-left of the fairness–utility trade-off plot.
  • VQA-Task-1 (LLaVA-1.5, PaliGemma on FACET, 15,623 images): ALA achieves competitive \(M_{RC}\) with near-zero \(D_{WCA}\); competing embedding-based methods show large negative \(D_{WCA}\) indicating subgroup accuracy loss.
  • VQA-Task-2 (LLaVA-1.5, PaliGemma on SocialCounterfactuals, 5,200 images): ALA reduces mean toxicity (\(D_{\text{mean}}\)) across gender, race, and physical-trait intersections while other methods trade utility for fairness.
  • VQA-Task-3 (utility preservation): ALA-BA correctly answers direct gender questions post-debiasing; blinding methods (DeAR, SFID) fail this task.
  • Extended evaluation on Qwen2.5-VL-3B-Instruct confirms ALA generalizes to instruction-tuned architectures where prompt engineering alone fails.
  • Quantitative tables are in appendices; headline figures are presented as scatter plots (Figure 4); specific numerical values are not reproduced in the main text excerpt.

Limitations

  • \(\beta_i\) scores are pre-computed from a fixed classifier and vocabulary, so they may not adapt to domain-shifted or novel token distributions at inference.
  • External classifiers (\(f^{\text{image}}\), \(f^{\text{text}}\)) are trained on separate datasets (FairFace, Bias-in-Bios, Wikipedia Toxicity); classifier quality directly bounds debiasing quality and may introduce its own biases.
  • Evaluation focuses on gender, race, and toxicity; coverage of other protected attributes (disability, religion, etc.) is not demonstrated.
  • The threshold \(\tau = 0.1\) and step size \(\lambda\) require per-task tuning; robustness to hyperparameter choice across diverse architectures is not systematically studied.
  • The text classifier call at each generation step (when \(|\beta_{i_t}| \geq \tau\)) still adds inference latency; no wall-clock benchmarking is reported.
  • The method addresses output-level bias but cannot correct biases in the model's internal reasoning that do not surface as explicit tokens with high \(|\beta_i|\).

Relevance to Vision-Language Models

ALA directly targets a critical fairness failure mode of VLMs—the propagation of societal biases into autoregressive image-to-text outputs—without requiring access to or modification of model weights, making it applicable to large proprietary or instruction-tuned VLMs (LLaVA, PaliGemma, Qwen2.5-VL). The logit-space intervention is architecturally orthogonal to the multimodal fusion strategy and can stack on top of RAG pipelines, addressing a robustness gap that representation-level methods leave open. For VLM researchers, the work establishes a clean new axis of post-hoc intervention and a reusable evaluation protocol (\(D_{WCA}\), VQA-Task-3) that decouples fairness from utility measurement. It also demonstrates that Integrated Gradients, typically used for interpretability, can serve as a practical debiasing primitive at inference time.