Lost in Visual Translation: A VLM-Assisted Perceptual-Semantic Coherence Framework for EEG-to-Image Reconstruction

🕒 Published (v1): 2026-07-14 05:40 UTC · Source: Arxiv · link

Why this paper was selected

VLM-assisted perceptual-semantic coherence for EEG-to-image; novel eval framework

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EEG-to-image reconstruction evaluation is broken: pixel and feature metrics either harshly penalize semantically recoverable outputs or reward perceptually plausible but semantically wrong ones. This paper introduces a VLM-assisted evaluation framework that uses four open-weight VLMs to annotate 6,855 reconstruction pairs with structured perceptual and semantic questions, then distills the consensus into BCI-Coherence Score (BCS), a lightweight reusable evaluator. BCS achieves T-PAS MAE 0.079 (\(r=0.700\)) and T-SAS MAE 0.082 (\(r=0.850\)), with human validation confirming Cohen's \(\kappa = 0.882\).

Problem

Standard reconstruction metrics (SSIM, LPIPS, CLIP similarity) were designed for high-fidelity image settings and fail in the low-SNR regime of EEG decoding. They exhibit two failure modes: (1) harshness — penalizing reconstructions that are visually degraded but semantically recoverable; and (2) semantic blindness — rewarding perceptually plausible reconstructions that depict the wrong object or scene. No single existing metric jointly tracks both semantic consistency and perceptual degradation reliably (MSE/SSIM show \(\rho_\text{cap} \approx -0.012\) and \(-0.044\) correlation with caption-level semantics).

Method

Metric failure analysis: 6,855 ground-truth/reconstruction pairs from ATM, ENIGMA, BrainVis, and DreamDiffusion are analyzed using (a) caption-level semantic probes (BLIP + SBERT cosine similarity), (b) quartile-based harshness/blind-spot rates, and (c) controlled degradation probes (six degradation types × four severity levels).

VLM annotation protocol: Four open-weight VLMs (InternVL3, SAIL-VL, OLA-7B, Ovis2-8B) evaluate each pair via 12 structured binary/ternary questions (\(\phi \in \{0, 0.5, 1\}\)): six perceptual (layout, shape, texture, color, artifacts, holistic visual recoverability) and six semantic (category, specific identity, function, quantity, scene context, holistic semantic recoverability). Per-pair scores are: $\(\text{T-PAS}_i^{(v)} = \frac{1}{6}\sum_{p=1}^{6} \phi(a_{i,p}^{(v)}), \quad \text{T-SAS}_i^{(v)} = \frac{1}{6}\sum_{s=1}^{6} \phi(a_{i,s}^{(v)})\)$ Consensus targets use the per-question median across VLMs, preserving fractional disagreement (\(z_{i,q} = \text{median}_{v}\,\phi(a_{i,q}^{(v)})\)).

BCS distillation: A lightweight MLP regression head is trained on frozen fusion features from SigLIP + CLIP + DINOv3. Pairwise comparison features concatenate embeddings, absolute differences, elementwise products, and cosine similarities. Training uses a weighted MSE loss where agreement weight \(w_{i,q} = 1 - \text{Var}_{v}\,\phi(a_{i,q}^{(v)})\) down-weights high-disagreement annotations. Split is 70/15/15 by semantic concept to prevent leakage.

Key Contributions

• Empirical analysis of 6,855 EEG reconstruction pairs showing pixel metrics are near-zero correlated with semantic consistency and no metric jointly tracks both semantic and perceptual properties.
• Quantified failure rates (Caption-HR, Caption-BSR) and controlled degradation probes formalizing the harshness/blind-spot taxonomy.
• A degradation-tolerant VLM annotation protocol producing T-PAS and T-SAS scores, with Fleiss' \(\kappa = 0.363\) overall and high agreement on concrete attributes (object shape \(\kappa = 0.418\), functional role \(\kappa = 0.499\)).
• BCI-Coherence Score (BCS): a distilled, reusable image-pair evaluator that eliminates the need for repeated VLM inference at evaluation time.
• Human validation showing joint coherence judgments are highly reliable (\(\kappa = 0.882 \pm 0.174\), \(\alpha = 0.882\)).

Results

• BCS best variant (4-VLM continuous ensemble, frozen SigLIP+CLIP+DINOv3 fusion):
• T-PAS MAE = 0.079, \(r = 0.700\)
• T-SAS MAE = 0.082, \(r = 0.850\)
• Question-level \(r = 0.815\), \(\rho = 0.798\)
• SigLIP-only baseline: PAS MAE = 0.098 (\(r=0.639\)), SAS MAE = 0.103 (\(r=0.782\))
• 4-VLM ordinal ensemble: PAS MAE = 0.086 (\(r=0.645\)), SAS MAE = 0.091 (\(r=0.828\))
• Conventional metric semantic correlations: OpenCLIP \(r_\text{cap} = 0.553\); DreamSim \(r_\text{cap} = 0.526\); SSIM \(r_\text{cap} = -0.059\); MSE \(r_\text{cap} = -0.003\)
• Strongest pairwise VLM agreement: InternVL3–Ovis2-8B, weighted \(\kappa = 0.594\), rationale similarity = 0.700

Limitations

• Analysis is restricted to four EEG-to-image models (ATM, ENIGMA, BrainVis, DreamDiffusion); generalization to other neural decoding modalities (e.g., fMRI) is not demonstrated.
• VLM annotation consensus shows only moderate overall Fleiss' \(\kappa = 0.363\); holistic recoverability questions are especially unstable (unanimous agreement as low as 4.0%).
• BCS is trained and evaluated on a single dataset of 6,855 pairs; cross-dataset or cross-subject generalization is not assessed.
• The approach requires VLM inference for annotation generation; the distilled BCS reduces but does not eliminate this cost for initial dataset labeling.
• Caption-based semantic probes (BLIP + SBERT) may miss fine-grained visual details, limiting the reliability of the semantic reference signal.

Relevance to Vision-Language Models

This paper demonstrates a concrete application of VLMs as structured, scalable annotators for domains where ground-truth quality is inherently ambiguous — directly relevant to VLM evaluation methodology and capability assessment. It shows that different open-weight VLMs (InternVL3, SAIL-VL, OLA-7B, Ovis2-8B) diverge significantly in calibration yet converge in rationale similarity, raising questions about VLM consistency on degraded visual inputs that are central to VLM robustness research. The BCS distillation pipeline — using frozen CLIP/SigLIP/DINOv3 encoders with pairwise comparison features — is a reusable recipe for building lightweight evaluators from VLM pseudo-labels, a technique broadly applicable to VLM-assisted annotation workflows. The structured question decomposition (perceptual vs. semantic recoverability) also provides a template for decomposed VLM evaluation beyond holistic similarity scores.