Skip to content

FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants

🕒 Published (v1): 2026-03-27 01:55 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; fairness-aware PEFT for multimodal LLMs in clinical high-stakes settings

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FairLLaVA is a parameter-efficient fine-tuning strategy that mitigates demographic bias in multimodal large language models by minimizing mutual information between the model's hidden states and sensitive attributes (gender, race, age). Applied to medical imaging tasks — chest radiology report generation and dermoscopy VQA — it reduces inter-group performance gaps without sacrificing overall clinical NLG quality. The method is architecture-agnostic and adds only ~57K parameters via a variational demographic attribute classifier (DAC) and LoRA adapters.

Problem

MLLMs exhibit uneven performance across demographic groups (gender, race, age) even when demographic labels are absent from text outputs, because medical images leak latent sensitive signals and acquisition pipelines correlate with demographics. Standard fairness techniques — frequency resampling, loss reweighting, adversarial classifiers — either fail to address intersectional cross-attribute dependencies, incur prohibitive inference cost, or cause catastrophic forgetting of clinical knowledge. Word-level (lexical) fairness probes from the NLP literature are inapplicable since radiology reports do not contain explicit demographic tokens. No prior work had studied bias mitigation specifically for open-ended generative MLLMs in clinical settings.

Method

FairLLaVA operates in two stages atop a LLaVA-style MLLM (Vicuna-7B backbone + BioMedCLIP encoder):

Stage 1 — Visual-language alignment: Only the multimodal projector \(\psi\) is fine-tuned with standard cross-entropy LM loss \(\mathcal{L}_{\text{LM}}\); vision encoder and LM are frozen.

Stage 2 — Fairness-aware fine-tuning via mutual information minimization: - A lightweight variational Demographic Attribute Classifier (DAC) \(\phi\) is trained simultaneously (not pre-trained) to predict attribute \(a\) from pooled hidden states \(h(x) = \frac{1}{|K|}\sum_{l \in K} h^l(x)\) across a tapped layer set \(K\), via: $\(\mathcal{L}_{\text{DAC}}(\phi) = -\mathbb{E}_{(a,x)\sim\mathcal{D}}\log\phi\!\left(a \mid h(x)\right)\)$ - The Demographic Information Minimization (DIM) loss minimizes an upper-bound surrogate of mutual information \(I^u(a, h(x))\), implemented via positive/negative pair contrasting: $\(\mathcal{L}_{\text{DIM}}(\theta,\psi) = \frac{1}{B}\sum_i \log\phi(a_i|h(x_i)) - \frac{1}{B(B-1)}\sum_{i,j\neq i}\log\phi(a_i|h(x_j))\)$ - Total loss: \(\mathcal{L}_{\text{total}} = \lambda_1 \mathcal{L}_{\text{DAC}} + \lambda_2 \mathcal{L}_{\text{DIM}} + \lambda_3 \mathcal{L}_{\text{LM}}\)

Training alternates: (1) update \(\phi\) only via \(\mathcal{L}_{\text{DAC}}\) with stop-gradient on \(h\); (2) freeze \(\phi\), update LoRA adapters on the LM and projector \(\psi\) via \(\mathcal{L}_{\text{DIM}} + \mathcal{L}_{\text{LM}}\).

The Equity-Scaled Metric (ES-M) is introduced for generative evaluation: \(\text{ES-M}_a = M_\text{all} / (1 + \Delta M_a)\), where \(\Delta M_a = \max_{a} M_a - \min_{a} M_a\) is the fairness gap, jointly rewarding high overall performance and low disparity.

Key Contributions

  • FairLLaVA: a plug-and-play MI-regularized LoRA fine-tuning method that produces demographic-invariant hidden representations without requiring pre-trained adversarial classifiers or demographic augmentation of the training data.
  • Extended Equity-Scaled Metric (ES-M) adapted for open-ended generative models: integrates any NLG metric with a fairness gap penalty into a single scalar.
  • Demonstration across two chest X-ray datasets (MIMIC-CXR, PadChest) and one RGB dermoscopy dataset (HAM10000), covering age, gender, and race, with multiple clinical and NLG metrics (BLEU-1/4, ROUGE-L, RadGraph-F1, GREEN score).
  • Empirical evidence that adversarial classifiers cause catastrophic forgetting in MLLMs, motivating the MI-based approach.

Results

  • On MIMIC-CXR (joint debiasing across all attributes, Tab. 1), FairLLaVA-All achieves 7 of 12 best ES-scores across Race, Age Group, and Gender dimensions, outperforming LLaVA-Rad, MedGemma-4B/27B, Qwen2.5-7B, DeepSeek-VL2, CheXagent, Reweighting-All, Resampling-All, and Adv. MLP Classifier-All.
  • Selected ES-scores for FairLLaVA-All on MIMIC-CXR (Race): ES-BLEU-1=13.36, ES-RadGraph-F1=6.93, ES-GREEN=4.06; all substantially above baselines (e.g., Adv. MLP Classifier: ES-BLEU-1=3.92, ES-GREEN=1.11).
  • On MIMIC-CXR per-attribute debiasing (Tab. 2), FairLLaVA achieves top ES scores on the targeted attribute with beneficial spillover; e.g., FairLLaVA-Race: ES-GREEN (Race)=3.24 vs. Reweighting-Race=2.99; FairLLaVA-Gender: ES-BLEU-1 (Gender)=25.43 vs. Reweighting-Gender=25.78.
  • DAC adds only ~57K parameters even on MIMIC-CXR with 14 attribute categories; training takes ~17 hours on 8× A6000 GPUs for MIMIC-CXR and ~4 hours for PadChest.

Limitations

  • Evaluation is restricted to medical imaging domains (radiology reports, dermoscopy); generalization to non-clinical VQA or general-purpose MLLMs is not demonstrated.
  • The MI upper-bound is estimated via a variational DAC; the tightness of this bound and sensitivity to hyperparameters \(\lambda_1, \lambda_2, \lambda_3\) is acknowledged but not fully ablated in the main paper (deferred to supplementary).
  • Race labels in MIMIC-CXR are coarsened to 4+1 groups, and reported gender is binary (Male/Female), limiting coverage of intersectional or non-binary demographic analysis.
  • The two-step alternating update (DAC then LoRA) requires careful training stability management; the paper notes supplementary experiments are needed to characterize this.
  • Only Vicuna-7B with BioMedCLIP is evaluated; the claimed architecture-agnosticism is not empirically verified on other backbones.

Relevance to Vision-Language Models

FairLLaVA directly addresses a practical deployment gap for medical VLMs: that image-conditioned generation amplifies demographic disparities through latent visual shortcuts invisible to word-level fairness probes. The MI-minimization regularizer over pooled transformer hidden states is broadly applicable to any transformer-based VLM fine-tuning pipeline, and the LoRA integration makes it low-cost enough to add to existing PEFT workflows. The extended ES-M metric fills a methodological void — prior fairness metrics assumed classification outputs, not generative text — and could become standard tooling for evaluating fairness in VLMs applied to clinical NLG. For VLM researchers, this work highlights that representation-level debiasing (rather than data resampling) is necessary when demographic signal is encoded in the visual modality rather than in lexical patterns.