What Images Cannot Say: Language-Guided Olfactory Representation Learning

• URL: http://arxiv.org/abs/2607.06402v1
• Date: Tue, 07 Jul 2026 15:31:55 GMT
• Title: What Images Cannot Say: Language-Guided Olfactory Representation Learning
• Abstract summary: We introduce SCENT, a framework that uses language guidance as a semantic bridge between vision and olfaction.<n>We train a smell encoder that maps electronic-nose signals into a shared embedding space aligned with both visual and textual representations.<n>Our results reveal the importance of contextual semantic information for grounding olfactory perception in multimodal learning.
• Score: 17.56125098945589
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Images tell us what a scene looks like, but rarely what it would feel like to be there. While recent datasets pair visual scenes with electronic-nose measurements, aligning smell signals with images remains challenging because many olfactory cues arise from contextual environmental factors that are not directly visible in pixels. We introduce SCENT, a multimodal framework that uses language guidance as a semantic bridge between vision and olfaction. Our approach leverages Vision-Language Models (VLMs) to generate scene descriptors capturing objects, environmental context, and plausible ambient smell cues suggested by the visual scene. These descriptors provide semantic guidance for learning olfactory representations. We train a smell encoder that maps electronic-nose signals into a shared embedding space aligned with both visual and textual representations, and introduce a languageguided latent decomposition that separates object-specific odors from contextual environmental contributions. Experiments on the New York Smells dataset demonstrate that SCENT significantly improves crossmodal retrieval compared to vision-only baselines, achieving state-of-theart performance on smell-to-image and smell-to-text retrieval tasks. In addition, our framework produces interpretable olfactory representations that enable the disentanglement of complex smell mixtures. Our results reveal the importance of contextual semantic information for grounding olfactory perception in multimodal learning and pave the way for future research in this area.

Related papers

• See & Sniff: Learning Visuo-Olfactory Representations [42.26043073031702]
  We introduce SmellNet-V, a scalable visuo-olfactory dataset built on the insight that odor identity is largely invariant to visual transformations within a semantic category.<n>This allows us to synthetically pair smell-only samples with semantically aligned in-the-wild web images, converting a unimodal olfactory dataset into a cross-modal benchmark.<n>We propose See & Sniff, a self-supervised framework that learns joint visuo-olfactory representations via dense local alignment.
  arXiv  Detail & Related papers  (2026-06-25T17:26:47Z)
• New York Smells: A Large Multimodal Dataset for Olfaction [93.162210916881]
  We present New York Smells, a large dataset of paired image and olfactory signals captured in the wild''<n>Our dataset contains 7,000 smell-image pairs from 3,500 distinct objects across indoor and outdoor environments.<n>Our benchmark has three tasks: cross-modal smell-to-image retrieval, recognizing scenes, objects, and materials from smell alone, and fine-grained discrimination between grass species.
  arXiv  Detail & Related papers  (2025-11-25T17:44:50Z)
• StableSemantics: A Synthetic Language-Vision Dataset of Semantic Representations in Naturalistic Images [5.529078451095096]
  understanding the semantics of visual scenes is a fundamental challenge in Computer Vision. Recent advancements in text-to-image frameworks have led to models that implicitly capture natural scene statistics. Our work presents StableSemantics, a dataset comprising 224 thousand human-curated prompts, processed natural language captions, over 2 million synthetic images, and 10 million attention maps corresponding to individual noun chunks.
  arXiv  Detail & Related papers  (2024-06-19T17:59:40Z)
• VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning [66.23296689828152]
  We leverage the capabilities of Vision-and-Large-Language Models to enhance in-context emotion classification.<n>First, we prompt VLLMs to generate natural language descriptions of the subject's apparent emotion in relation to the visual context.<n>Second, the descriptions, along with the visual input, are used to train a transformer-based architecture.
  arXiv  Detail & Related papers  (2024-04-10T15:09:15Z)
• CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations [98.30038910061894]
  Vision-and-Language Navigation (VLN) tasks require an agent to navigate through the environment based on language instructions. We propose CLEAR: Cross-Lingual and Environment-Agnostic Representations. Our language and visual representations can be successfully transferred to the Room-to-Room and Cooperative Vision-and-Dialogue Navigation task.
  arXiv  Detail & Related papers  (2022-07-05T17:38:59Z)
• SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning [61.57887011165744]
  multimodal Transformers have made great progress in the task of Visual Commonsense Reasoning. We propose a Scene Graph Enhanced Image-Text Learning framework to incorporate visual scene graphs in commonsense reasoning.
  arXiv  Detail & Related papers  (2021-12-16T03:16:30Z)
• Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization [81.26077816854449]
  We first explore the use of constituency parse trees for encoding structured input. Second, we augment the structured input with commonsense information and study the impact of this external knowledge on the generation of visual story. Third, we incorporate visual structure via bounding boxes and dense captioning to provide feedback about the characters/objects in generated images.
  arXiv  Detail & Related papers  (2021-10-21T00:16:02Z)
• COBE: Contextualized Object Embeddings from Narrated Instructional Video [52.73710465010274]
  We propose a new framework for learning Contextualized OBject Embeddings from automatically-transcribed narrations of instructional videos. We leverage the semantic and compositional structure of language by training a visual detector to predict a contextualized word embedding of the object and its associated narration. Our experiments show that our detector learns to predict a rich variety of contextual object information, and that it is highly effective in the settings of few-shot and zero-shot learning.
  arXiv  Detail & Related papers  (2020-07-14T19:04:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.