Knowing You at First Glance: Inferring Apparent Personality from Faces¶

🕒 Published (v1): 2026-07-16 06:48 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

GlanceFace is an end-to-end framework that infers apparent MBTI personality types from a single facial image, requiring no conversational or audio signals. It combines VLM-derived semantic priors with a differential gated attention module and an uncertainty-aware loss to handle noisy, subjective annotations. The system is motivated by the need for embodied agents to form an initial user model before any interaction begins.

Problem¶

Prior apparent personality inference systems rely on multimodal inputs (audio, video, text) that are only available after interaction starts. No robust face-only system exists for MBTI prediction—a more structured and LLM-interpretable personality schema than the Big Five. Two core challenges compound the difficulty: (1) personality-relevant facial cues are weak, implicit, and fine-grained, and (2) crowdsourced personality annotations are inherently noisy and subjective, degrading supervised training.

Method¶

Backbone: Modified ResNet-18 (IR-18) processes $112 \times 112$ aligned face crops into hierarchical feature maps.

Semantic-Enhanced Facial Representation (SEFR): Qwen3-VL-Embedding-2B extracts a 2048-dim global semantic embedding $E_V$. Because $E_V$ is a flat vector (not a token sequence) and may contain noise, a differential gated attention mechanism computes: $$DA = \left(\text{Softmax}\!\left(\frac{Q_0 \odot K_0}{\sqrt{d}}\right) - \text{Softmax}\!\left(\frac{Q_1 \odot K_1}{\sqrt{d}}\right)\right) \odot V$$ The difference $A_{\text{diff}}$ suppresses shared/uninformative activations. A subsequent gated FFN ($\text{GELU}$ gate) further refines the signal. One SEFR block is inserted after each ResNet block, modulating features via: $$F_i^{\text{fuse}} = F_i^{RN} + F_i^{RN} \odot \text{Tanh}(F_i^g)$$

Uncertainty-Aware Personality Learning (UAPL): For each sample the dataset provides a soft vote distribution $S_T^i(t) \in \mathbb{R}^2$ and a vote count $N_{\text{vote}}^i$. UAPL weights each sample by a combined score $w^i(t) = s_t^i \cdot w_{\text{vote}}^{i\,\alpha} \cdot w_{\text{ent}}^{i\,\beta}(t)$, where $w_{\text{vote}}$ is a log-normalized vote count, $w_{\text{ent}}$ is the complement of normalized Shannon entropy of the soft label, and $s_t^i$ is a hard confidence gate ($\tau=0.75$) that zeroes out highly ambiguous samples. The loss matches predictions to soft targets via Jensen-Shannon divergence plus a $\ell_1^\gamma$ margin term, weighted by reliability, plus an AdaFace identity regularizer $L_{FR}$.

Inference: Four independent binary heads predict each MBTI dimension; the 16-type probability is the product of the four marginals $p_k^{(\text{mbti})} = \prod_{d=1}^4 P(X_d = x_d^{(k)})$, giving Top-$K$ rankings.

Key Contributions¶

First systematic face-only framework targeting MBTI apparent personality prediction, framed as a prior-to-interaction tool for embodied agents.
SEFR: differential gated attention over a global VLM embedding, progressively injected into hierarchical CNN features to amplify subtle personality-relevant cues.
UAPL: a unified uncertainty-aware loss combining vote-based reliability, entropy-based confidence gating, Jensen-Shannon soft matching, and $\ell_1^\gamma$ margin penalty.
Correlation analysis via Grad-CAM++ showing that opposing MBTI traits differ in activation emphasis over shared facial regions (eyes/nose/mouth) rather than entirely separate spatial areas; F-T pair shows lowest inter-trait correlation (0.84).

Results¶

All numbers are from the MBTI dataset (3,092 training / 216 evaluation identities; ~568K / ~39K images).

16-type MBTI classification (person-level): - GlanceFace Acc@1: 26.39%, Acc@3: 56.48%, Acc@5: 75.00%, F1: 17.99%, AUC: 78.19% - Best baseline (AdaFace IR-18): Acc@1: 21.30%, F1: 14.28%, AUC: 75.78%

16-type MBTI classification (image-level): - GlanceFace Acc@1: 18.84%, AUC: 68.91% vs. AdaFace: 15.53% / 64.31%

4-dimension binary classification (person-level average): - GlanceFace Acc: 71.41%, F1: 73.18%, AUC: 74.19% vs. AdaFace: 68.98% / 71.00% / 71.48%

Ablation (person-level 16-MBTI Acc@1): - IR-18 baseline: 18.98% → +SEFR: 19.44% → +UAPL: 23.61% → both: 26.39%

Limitations¶

The task measures apparent (perceived) personality, not ground-truth personality; the model learns social stereotypes embedded in human annotations, not objective traits.
Dataset labels are sourced from a single external personality database (Personality Database), which may introduce selection and cultural biases not controlled for.
Evaluation is limited to one dataset; cross-dataset or cross-cultural generalization is untested.
The 16-type Acc@1 of 26.39% against random chance of 6.25% is a real signal but still leaves substantial prediction error for deployment.
Ethical risks of using facial appearance to infer personality are acknowledged but not addressed beyond noting the model reflects "perceived" rather than "real" traits.
The paper's text is truncated before the conclusion; any additional limitations stated there are not captured.

Relevance to Agentic AI / LLM Agents¶

The explicit motivation is to give embodied agents a cold-start mechanism: before any speech or interaction, an agent can hypothesize a user's personality type and pre-configure its dialogue or behavioral strategy accordingly. This directly addresses the initialization problem in human-robot interaction and aligns with lines of work building adaptive agent personas. The MBTI framing is chosen specifically because MBTI types are natively interpretable by LLMs, making the output of GlanceFace a structured prior that can be injected into an LLM agent's system prompt or memory. For researchers tracking agentic AI, the key takeaway is that vision–language embedding models can serve as a lightweight perception front-end that enriches an agent's world model from a single sensory frame.