What Images Cannot Say: Language-Guided Olfactory Representation Learning¶

🕒 Published (v1): 2026-07-07 15:31 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SCENT (Semantic Context-aware e-Nose Transformer) uses VLM-generated language descriptors—capturing visible objects, environmental context, and inferred ambient smells—as a semantic bridge to align electronic-nose (e-nose) signals with vision and text embeddings. This addresses a fundamental failure mode of vision-only supervision: olfactory signals diffuse freely from sources outside the camera's field of view, so images provide only partial observability. SCENT achieves state-of-the-art cross-modal retrieval on the New York Smells benchmark while yielding interpretable, disentangled smell representations.

Problem¶

E-nose signals capture ambient chemical compositions that can originate from sources entirely outside the camera frame—meaning visual supervision alone is structurally insufficient for olfactory representation learning. Existing work (NYS baseline) aligns smell with images only, missing latent olfactory cues implied by scene context. Additionally, smell recordings are mixtures from multiple odor sources (object vs. environmental background) that prior methods do not attempt to disentangle.

Method¶

SCENT operates in three stages:

Stage 1 — VLM Scene Augmentation. A frozen Qwen3VL-30B model is prompted to generate structured descriptors for each image: $$\mathcal{T}(I) = \{t_{\text{obj}},\, t_{\text{ctx}},\, t_{\text{smell}}^{(1)},\ldots,t_{\text{smell}}^{(K)}\}$$ covering the primary object, surrounding environmental context, and $K$ plausible ambient smell cues inferred from scene semantics. These are concatenated into a single prompt and encoded by the frozen CLIP ViT-B/16 text encoder into $z^T \in \mathbb{R}^{512}$.

Stage 2 — Multimodal Alignment. A Transformer smell encoder (6 layers, 8 heads) maps the 32-channel × $2T$-timestep e-nose signal $X$ to $z^S$. Two MLP projection heads produce $z^S_I$ and $z^S_T$ targeting visual and textual CLIP spaces, respectively. The model is trained with symmetric InfoNCE losses $\mathcal{L}_{IS}$ and $\mathcal{L}_{ST}$: $$\mathcal{L}_{\text{total}} = \lambda_{IS}\mathcal{L}_{IS} + \lambda_{ST}\mathcal{L}_{ST}$$ The CLIP image encoder is fine-tuned; the text encoder is frozen.

Stage 3 — Latent Disentanglement. Two additional projection heads decompose $z^S_T$ into object-specific ($z^S_{\text{obj}}$) and contextual ($z^S_{\text{ctx}}$) components, each contrastively aligned to its corresponding CLIP text descriptor ($t_{\text{obj}}$, $t_{\text{ctx}}$). A reconstruction loss enforces that the concatenated latents reconstruct $X$: $$\mathcal{L}_{\text{dis}} = \mathcal{L}_{\text{obj}} + \mathcal{L}_{\text{ctx}} + \lambda_{\text{rec}}\|X - d([z^S_{\text{obj}}, z^S_{\text{ctx}}])\|^2$$

Key Contributions¶

Language-guided semantic augmentation: VLM-inferred smell descriptors expose olfactory context invisible to cameras, providing supervision beyond visual partial observability.
SCENT framework: joint trimodal alignment of e-nose, CLIP image, and CLIP text embeddings via dual InfoNCE objectives.
Language-guided latent decomposition: contrastive disentanglement of object-level vs. environmental odor contributions, regularized by signal reconstruction.
Demonstration that VLM world-knowledge priors transfer to a non-visual sensory modality (machine olfaction), establishing a novel precedent.

Results¶

All results on the New York Smells (NYS) dataset (936 validation samples, 7,000 total image-smell pairs across 3,500 object categories).

Semantic ablation (frozen backbones): Image-only baseline R@5 = 12.1; object-label-only text alignment R@5 = 8.8 (underperforms); adding context raises to 10.8; adding inferred smells to 12.4; full O+Ctx+S with joint Image&Text alignment reaches R@5 = 14.3, R@10 = 22.5, R@20 = 35.1.
S2I (Smell-to-Image): SCENT 23.0 R@5 vs. NYS baseline 20.0 R@5.
S2T (Smell-to-Text): SCENT 11.9 R@5, a 47% relative improvement over the adapted NYS "Image Bridge" baseline.
S2IT (joint Smell-to-Image+Text): SCENT outperforms the adapted baseline by >7% in R@5.
Cosine similarity analysis confirms SCENT's olfactory embedding is more discriminative across retrieval ranks than NYS; visual features are near-flat across ranks and insufficient for the task.

Limitations¶

Evaluation is on a single benchmark (NYS); generalization to other e-nose hardware or non-urban olfactory domains is untested.
The 32-channel sensor array is hardware-specific; the encoder architecture may not transfer directly to other e-nose configurations.
VLM-generated smell descriptors are heuristic inferences and may be semantically inaccurate for visually ambiguous or atypical scenes—this noise is not explicitly quantified.
The paper is truncated; ablation results on disentanglement quality and sensitivity to $\lambda_{\text{rec}}$ are not fully available in the provided text.
The method requires running a 30B-parameter VLM (Qwen3VL-30B) at annotation time, adding significant preprocessing cost versus vision-only baselines.

Relevance to Vision-Language Models¶

This paper is a direct application of VLMs as cross-modal knowledge injectors: rather than using VLMs for image captioning or VQA, SCENT mines their world-knowledge priors to generate olfactory supervision that bridges the partial-observability gap inherent in vision-based smell learning. It demonstrates that VLM representations encode implicit sensory associations (smell, texture, etc.) beyond vision, which aligns with recent interest in grounding VLMs to non-standard modalities (tactile, audio). The use of CLIP's frozen text encoder as a shared semantic anchor for aligning a non-visual sensor modality is a methodologically clean extension of the CLIP/ImageBind paradigm to machine olfaction. Researchers tracking VLMs will find this a compelling demonstration that VLM semantic embeddings generalize as supervisory signals even where visual input is structurally insufficient.