Skip to content

Knowing You at First Glance: Inferring Apparent Personality from Faces

🕒 Published (v1): 2026-07-16 06:48 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GlanceFace is an end-to-end framework that infers apparent MBTI personality types from static facial images alone by fusing VLM-derived semantic embeddings with hierarchical CNN face features. A differential gated attention module (SEFR) selectively injects these priors into the backbone, while an uncertainty-aware loss (UAPL) mitigates noisy and subjective personality annotations. The framework achieves state-of-the-art MBTI prediction under both person-level and image-level protocols.

Problem

Existing apparent personality inference systems depend on multimodal signals (audio, video, text) that are only available after interaction has begun, making them inapplicable in the prior-to-interaction setting where only a face image is accessible. MBTI-based personality prediction from faces alone is underexplored, and the task is doubly hard: associations between facial appearance and perceived MBTI traits are weak and implicit, and crowd-sourced personality annotations are inherently noisy and subjective.

Method

Backbone & VLM priors. A modified ResNet-18 extracts hierarchical facial feature maps \(\{F_i^{RN}\}\). In parallel, Qwen3-VL-Embedding-2B produces a 2048-dimensional global semantic embedding \(E_V \in \mathbb{R}^{D_V}\) per image.

Semantic-Enhanced Facial Representation (SEFR). Because \(E_V\) is a flat vector (not a token sequence) and may contain task-irrelevant noise, a lightweight differential gated attention is applied element-wise: $\(DA = \left(\text{Softmax}\!\left(\frac{Q_0 \odot K_0}{\sqrt{d}}\right) - \text{Softmax}\!\left(\frac{Q_1 \odot K_1}{\sqrt{d}}\right)\right) \odot V\)$ The differential response \(A_{diff}\) suppresses shared activations and highlights discriminative semantic cues. A subsequent gated FFN further refines the result: $\(F_{mid} = \text{LN}(W_{down} DA \odot \text{GELU}(W_{gate} DA)),\quad F_{Qw} = \text{LN}(W_{up} F_{mid})\)$ One SEFR module is inserted after each ResNet block; the refined VLM features modulate the ResNet feature map via channel-wise Tanh gating: \(F_i^{fuse} = F_i^{RN} + F_i^{RN} \odot \text{Tanh}(F_i^g)\).

Uncertainty-Aware Personality Learning (UAPL). Each training sample carries a soft label distribution \(S_T^i(t) \in \mathbb{R}^2\) and a vote count \(N_{vote}^i\) for each of the four MBTI dimensions. UAPL defines a composite per-sample weight combining (i) log-normalized vote reliability \(w_{vote}^i\), (ii) complement-of-entropy confidence \(w_{ent}^i(t)\), and (iii) a hard-discard indicator \(s^i_t\) that zeroes out samples with \(\max(S_T) \le \tau = 0.75\): $\(w^i(t) = s^i_t \cdot (w_{vote}^i)^\alpha \cdot (w_{ent}^i(t))^\beta\)$ Prediction is trained by a weighted combination of Jensen-Shannon divergence against soft targets, an \(L^1\) margin penalty, and AdaFace face-recognition regularization \(L_{FR}\). At inference, each of the four binary MBTI dimension probabilities is multiplied to rank all 16 types (factorized formulation).

Key Contributions

  • GlanceFace framework: first end-to-end system targeting face-only apparent MBTI personality inference in the prior-to-interaction setting.
  • SEFR module: differential gated attention over global VLM embeddings enabling noise-aware, progressive injection of semantic priors into CNN feature hierarchies without requiring token-level VLM outputs.
  • UAPL strategy: uncertainty-aware loss that jointly leverages vote-count reliability, annotation entropy, soft-label JS divergence, and a hard-ambiguity filter to stabilize training under subjective labels.
  • Correlation analysis: Grad-CAM++ visualizations reveal that apparent personality differences manifest as subtle activation shifts within shared facial regions (eyes, nose, mouth) rather than spatially disjoint areas; opposing traits within a dimension show lower response-map correlation (F-T: 0.84) than cross-dimensional pairs.

Results

All comparisons are against face-backbone baselines (IR-18, IR-50, MobileFaceNet, ViT-S) and face-recognition models (CosFace, ArcFace, AdaFace) on the proposed MBTI dataset (3,092 train / 216 eval identities; ~568k / 40k images).

16-type MBTI classification (person-level): - GlanceFace: Acc@1 = 26.39%, Acc@3 = 56.48%, Acc@5 = 75.00%, F1 = 17.99%, AUC = 78.19% - Best baseline (AdaFace): Acc@1 = 21.30%, Acc@3 = 52.31%, Acc@5 = 67.13%, F1 = 14.28%, AUC = 75.78%

16-type MBTI classification (image-level): - GlanceFace: Acc@1 = 18.84%, Acc@3 = 43.90%, Acc@5 = 62.54%, F1 = 12.60%, AUC = 68.91% - Best baseline (MobileFace): Acc@1 = 16.00%, Acc@3 = 38.98%, Acc@5 = 56.06%

4-dimension binary prediction (person-level avg): - GlanceFace: Acc = 71.41%, F1 = 73.18%, AUC = 74.19% vs. AdaFace (68.98%, 71.00%, 71.48%)

Ablation (person-level 16-type Acc@1): - IR-18 baseline: 18.98% → +SEFR: 19.44% → +UAPL only: 23.61% → both: 26.39%

Limitations

  • MBTI dimensions are predicted independently (factorized); inter-dimension correlations are not modeled, which may underestimate joint personality structure.
  • Apparent personality inference reflects human perception biases, not validated psychological traits; model interpretability claims are constrained to learned response patterns, not causal links.
  • Trait-specific facial response maps are highly correlated (\(> 0.90\) for most pairs), suggesting the model captures a shared low-dimensional facial signal rather than truly trait-specific features.
  • The annotation source (Personality Database) introduces selection bias; crowd annotators may systematically disagree with MBTI self-reports.
  • Evaluated on a single proprietary dataset; generalization to other demographics, imaging conditions, or MBTI annotation protocols is unverified.

Relevance to Vision-Language Models

GlanceFace demonstrates a concrete pattern for extracting utility from a frozen VLM (Qwen3-VL-Embedding-2B) when the downstream task requires subtle, high-level visual semantics that go beyond standard visual features but do not naturally surface in auto-regressive VLM outputs. The SEFR module generalizes the idea of using a global VLM embedding as a semantic prior injected progressively into a CNN hierarchy, a design pattern applicable to any fine-grained facial or visual attribute task. The work also highlights a gap in VLM research: while VLMs are strong at open-vocabulary recognition, their embeddings contain task-irrelevant noise for specialized soft attributes, motivating differential filtering mechanisms. For researchers tracking VLMs, this paper offers a case study of "VLM-as-feature-extractor" for personality-related social perception, a domain where purely language-prompted VLMs struggle with weak visual-semantic grounding.