Skip to content

Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verification

🕒 Published (v1): 2026-03-26 16:30 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; demographic fairness benchmark for MLLM-based face verification

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks nine open-source MLLMs (2B–8B parameters) on face verification fairness across gender and ethnicity groups using IJB-C and RFW protocols. Results show that face-specialized FaceLLM-8B substantially outperforms general-purpose VLMs on accuracy, but MLLM bias patterns differ fundamentally from classical face recognition—Caucasian faces are most disadvantaged on IJB-C (opposite of typical embedding-based findings), while African faces are most disadvantaged on RFW. Accuracy and fairness do not co-vary: the most accurate model is not the fairest, and poor models can appear fair through uniformly high error rates.

Problem

Demographic fairness of MLLMs used as face verification systems is entirely unstudied. Prior work establishes that embedding-based face recognition exhibits demographic bias, but MLLMs process face images through visual question answering rather than learned identity embeddings, so their bias structure is unknown. Existing MLLM face benchmarks (FaceXBench, FaceRecBench) measure accuracy on face tasks but ignore fairness across ethnicity and gender subgroups.

Method

Nine open-source MLLMs from six families (Idefics3-8B, Ovis1.5-8B, Qwen2-VL-{2B,7B}, Qwen2.5-VL-{3B,7B}, Valley2-7B, LLaVA-NeXT-Mistral-7B, FaceLLM-8B) are evaluated zero-shot. Each model is prompted with a pair of face images and asked: "On a scale from 0 to 100, how likely (as a single number) are these two face images of the same person?" The response is normalized to \([0,1]\) as a similarity score.

For multi-image templates (IJB-C), the template-level score averages all \(m \times n\) pairwise scores: $\(s(T_p, T_g) = \frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n} s_{ij}\)$

Verification accuracy is measured via Equal Error Rate (EER) and True Match Rate (TMR) at FMR \(\in \{10\%, 1\%, 0.1\%\}\). Fairness is quantified with four FMR-based metrics per demographic group at the global EER threshold and the same three operating points: - \(\Delta_F\): max minus min per-group FMR (absolute differential) - \(R_F\): max-to-min per-group FMR ratio - \(M_F\): max per-group FMR divided by geometric mean across groups - \(G_F\): Gini coefficient of per-group FMR values

The decidability index \(d' = (\mu_\text{genuine} - \mu_\text{impostor}) / \sqrt{(\sigma_\text{genuine}^2 + \sigma_\text{impostor}^2)/2}\) is also reported per group. Evaluation uses two protocols: IJB-C (subsampled to 10,000 template comparisons, with ethnicity labels {African, East Asian, South Asian, Caucasian} and gender labels) and RFW (24,000 balanced pairs across {African, Asian, Caucasian, Indian}, no gender annotations).

Key Contributions

  • First systematic demographic fairness benchmark of MLLMs for face verification, covering nine models across six families.
  • Adoption of four complementary ISO/IEC 19795-10:2024-aligned FMR-based fairness metrics evaluated at multiple operating points, revealing that metric choice materially changes fairness conclusions.
  • Empirical finding that MLLM bias patterns differ qualitatively from embedding-based systems: Caucasian is the most disadvantaged ethnicity on IJB-C for five of eight models, in contrast to the dark-skin penalty typical of classical face recognition.
  • Identification of the accuracy–fairness trade-off: high-accuracy models are not necessarily the most equitable; low-accuracy models can appear fair due to uniformly high error rates across all groups.
  • Characterization of template size as a key performance driver: IJB-C's mean of 20.3 images/template vs. RFW's 3.6 explains the large performance gap across benchmarks.

Results

  • FaceLLM-8B (IJB-C): global EER 5.13%, TMR@FMR=10% of 98.0%; \(d'\) mean 4.50 — by far the best on accuracy.
  • Best general-purpose model (IJB-C): Qwen2-VL-7B at EER 8.54%; Qwen2.5-VL-7B at 10.43%.
  • Weakest models: Ovis1.5-8B (30.89% EER) and Qwen2-VL-2B (33.69% EER) on IJB-C, near chance.
  • RFW degradation: all models perform substantially worse; FaceLLM-8B rises to 29.46% EER; several competitive IJB-C models cluster around 35% EER; Ovis1.5 and Qwen2-VL-2B approach 50% (random chance). FaceLLM-8B \(d'\) drops from 4.50 to 1.08.
  • Ethnicity bias (IJB-C): Caucasian group has the highest per-group EER in 5/8 models — unusual vs. classical recognition. FaceLLM-8B: \(\Delta_F = 0.020\) (lowest absolute differential) but \(G_F = 0.092\) (higher than Qwen2-VL-2B's \(G_F = 0.059\)).
  • Ethnicity bias (RFW): African group most disadvantaged across all eight models. FaceLLM-8B has 35.25% EER on African vs. 21.23% on Asian — a 14 pp gap; \(\Delta_F = 0.478\), \(G_F = 0.293\) (highest among models).
  • Gender bias: systematically smaller than ethnicity bias. FaceLLM-8B: \(\Delta_F = 0.001\), \(G_F = 0.004\) at EER threshold. Maximum gender Gini across all models at FMR=10% is 0.123, vs. 0.172 for ethnicity.
  • LLaVA-NeXT-Mistral-7B produces only two unique similarity scores and is excluded from fairness analysis.

Limitations

  • Subsampling IJB-C to 10,000 comparisons (from 19,557) due to GPU cost (~20 GPU-days on H100 per model) limits coverage; scaling to the full benchmark is noted as challenging.
  • RFW lacks gender annotations, preventing gender fairness analysis on that dataset.
  • All evaluations are zero-shot; no fine-tuning or in-context examples are tested, so fairness under adaptation is unknown.
  • Only open-source models evaluated; proprietary frontier models (GPT-4V, Gemini) are excluded.
  • Models range only from 2B to 8B parameters; the study cannot speak to fairness of larger-scale MLLMs.
  • IJB-C ethnicity and gender labels were assigned by the authors, not from the original dataset release, introducing potential label noise.
  • The paper does not propose fairness mitigations; it is purely a diagnostic study.

Relevance to Vision-Language Models

This study directly probes whether general-purpose VLMs can substitute for specialized biometric systems and whether their bias structures mirror those of classical computer vision pipelines — both open questions of high relevance to the VLM research agenda. The finding that MLLM bias patterns diverge from embedding-based recognition (e.g., Caucasian disadvantage on IJB-C) suggests that VLMs encode demographic information in qualitatively different ways than task-specialized models, likely reflecting the composition of their pretraining data and the absence of identity-discriminative supervision. The accuracy–fairness decoupling and the sensitivity of fairness conclusions to metric choice are practical lessons for anyone deploying VLMs in safety-critical settings. This work also highlights the need for fairness-aware evaluation protocols alongside standard accuracy benchmarks whenever VLMs are adapted to high-stakes perception tasks.