Skip to content

AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memes

🕒 Published (v1): 2026-07-05 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This CLEF 2026 shared-task paper addresses multimodal sexism detection in memes by casting three hierarchically nested subtasks as conditional soft-label prediction over empirical annotator distributions. A lightweight Gated MLP (3.5M trainable parameters) maps frozen Gemini Embedding 2 vision-language representations to per-task soft distributions trained with KL divergence and homoscedastic uncertainty weighting. The system ranked 1st on the fine-grained sexism categorization subtask (Task 2.3) and 4th on the binary identification and source-intention subtasks on the official Soft-Soft leaderboards.

Problem

Prior multimodal hate/sexism detectors (CLIP-based fusion, LLM prompting) share two gaps: they collapse annotation disagreement into a majority label, erasing minority perspectives in a fundamentally subjective task; and they ignore the hierarchical dependency structure among subtasks (sexism identification → source intention → category assignment). The LeWiDi paradigm demands full annotator-distribution prediction rather than a hard label, requiring systems trained and evaluated differently from standard classifiers.

Method

Backbone. A frozen Gemini Embedding 2 encoder \(f_{\text{VLM}}: \mathcal{X} \to \mathbb{R}^{768}\) produces a fixed multimodal embedding \(\mathbf{x}\) per meme. This passes through one SwiGLU block with expansion factor 2 and dropout 0.2: $\(\text{SwiGLU}(\mathbf{x}) = (\mathbf{x}W_1) \odot \text{SiLU}(\mathbf{x}W_2), \quad \mathbf{h}^{(l+1)} = \mathbf{h}^{(l)} + W_3\,\text{SwiGLU}(\text{LN}(\mathbf{h}^{(l)}))\)$ Three independent linear heads produce logits for each subtask (binary sexism, binary intention, 5-way multi-label category).

Soft-label objective. Each head is trained against empirical annotator soft targets via KL divergence. For Task 2.3, per-category binary KL is averaged across the five sexism types.

Dynamic multi-task weighting. Homoscedastic uncertainty weighting introduces learnable variance parameters \(\sigma_i\) per task: $\(\mathcal{L}(W,\sigma_1,\sigma_2,\sigma_3) = \sum_{i=1}^{3}\left(\frac{1}{2\sigma_i^2}\mathcal{L}_i + \log\sigma_i\right)\)$ Losses for Tasks 2.2 and 2.3 are masked to sexist instances only, preventing gradient contamination from non-sexist examples.

Hierarchical decoding. At inference, joint distributions enforce the task ontology: Task 2.2 joint probability is \(\{\hat{p}^{(1)}\hat{p}^{(2)},\, \hat{p}^{(1)}(1-\hat{p}^{(2)}),\, 1-\hat{p}^{(1)}\}\); Task 2.3 per-category probabilities are \(\hat{p}^{(1)}\hat{p}^{(3)}_c\).

Ablation (Run 3). Detached soft-gating scales downstream representations by the sexism prediction with stopped gradients: \(\mathbf{h}_{\text{down}} = \mathbf{h} \odot \text{sg}[\hat{p}^{(1)}]\).

Key Contributions

  • Frozen VLM embeddings (Gemini Embedding 2) fed into a 3.5M-parameter Gated MLP, avoiding fine-tuning large generative models on small, noisy data.
  • KL divergence soft-label training that directly optimizes for the LeWiDi evaluation metric rather than cross-entropy against majority labels.
  • Homoscedastic uncertainty weighting that self-balances three heterogeneous multi-task losses without manual tuning.
  • Hierarchical conditional masking and probabilistic joint-probability decoding that enforce the task taxonomy without dedicated architectural branches.
  • Ablation demonstrating that explicit detached soft-gating (Run 3) yields negligible gain over conditional loss masking alone.
  • Quantitative EEG separability analysis (PCA + MANOVA) showing physiological signals provide insufficient linear discriminative signal for any subtask (Wilks' \(\lambda > 0.99\), all channels non-significant after Bonferroni correction).

Results

  • Task 2.3 (Soft-Soft): Ranked 1st overall; best run ICM-Soft = −2.8881, ICM-Soft Norm = 0.3469.
  • Task 2.1 (Soft-Soft): Ranked 4th; best run ICM-Soft = 0.2323, ICM-Soft Norm = 0.5373.
  • Task 2.2 (Soft-Soft): Ranked 4th; best run ICM-Soft = −0.6720, ICM-Soft Norm = 0.4285.
  • Hard-Hard (no threshold tuning): Task 2.1 rank 14 (ICM-Hard Norm 0.636), Task 2.2 rank 12 (0.501), Task 2.3 rank 8 (0.418) — substantially lower than Soft-Soft ranks, attributed to untuned 0.5 threshold.
  • Local test split: Run 1 Task 2.1 F1(YES) = 0.752, Run 2 Task 2.3 ICM-S Norm = 0.336; seed variance is notable across downstream tasks.
  • No physiological (EEG) signal survived Bonferroni correction (\(\alpha = 6.25 \times 10^{-4}\)) in MANOVA over 80 bandpower features reduced to 21 PCA components.

Limitations

  • Depends on proprietary Gemini Embedding 2; full pipeline reproducibility is constrained without access to the same API.
  • Hard-label decoding uses a fixed 0.5 threshold with no task-specific calibration, meaning Hard-Hard results are a lower bound on achievable performance.
  • EEG analysis restricted to linear separability; gaze and heart-rate signals excluded a priori rather than empirically, so their utility is unknown.
  • Small dataset (3,984 training memes) causes meaningful seed variance, especially on conditional downstream tasks.
  • Soft-gating ablation tested only as a full architectural variant; partial or annealed gating schedules were not explored.

Relevance to Vision-Language Models

This paper demonstrates a practical pattern for leveraging frozen VLM embeddings (Gemini Embedding 2) in specialized, low-data classification tasks where fine-tuning large models is inadvisable — a direct design choice relative to CLIP, SigLIP, and prompted Claude/LLM baselines. The soft-label training paradigm (KL divergence against annotator distributions) is directly applicable to any VLM-powered annotation or content-moderation pipeline where human label disagreement carries signal. The hierarchical conditional architecture also generalizes to other VLM tasks with ontological output dependencies (e.g., scene graph prediction, hierarchical image classification). The EEG separability negative result is a useful datapoint for VLM researchers considering physiological signal fusion with multimodal content embeddings.