Connecting Speech to Words through Images

• URL: http://arxiv.org/abs/2606.16807v1
• Date: Mon, 15 Jun 2026 14:50:42 GMT
• Title: Connecting Speech to Words through Images
• Abstract summary: We present a visually grounded method for building a vocabulary of spoken words using only images and their spoken descriptions.<n>In spoken word retrieval and keyword spotting experiments, the proposed approach outperforms a strong neural baseline while being more interpretable.
• Score: 23.312539472103566
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How can we learn the mapping between written words and their spoken counterparts in the absence of explicit textual supervision? We present a visually grounded method for building a vocabulary of spoken words using only images and their spoken descriptions. First, image captioning systems are used to build a vocabulary of written words representing salient visual concepts in the images. For each word, we then find utterances whose image captions contain that word. Then we use an unsupervised word discovery technique to align these utterances to locate instances of the target word. The result is spoken word segments that are linked to written words -- all accomplished without any text supervision. In spoken word retrieval and keyword spotting experiments, the proposed approach outperforms a strong neural baseline while being more interpretable. These results demonstrate the feasibility of the approach in English and motivate future work on low-resource languages without transcripts.

Related papers

• Do speech foundation models really learn words? [4.9931954371955465]
  Good discrimination of words may be explained by good encoding of word form (phonemes)<n>We show that, in later layers, HuBERT and wav2vec 2.0 do in general learn representations which encode words with reasonable fidelity independently of local phonetic content.
  arXiv  Detail & Related papers  (2026-09-09T16:47:59Z)
• Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding [23.312539472103566]
  We consider how we can map a written English keyword to spoken realisations of that word in Hindi.<n>Our work demonstrates that cross-lingual word-to-speech mappings can be learned directly from visual grounding.
  arXiv  Detail & Related papers  (2026-08-27T10:22:07Z)
• Text-to-Image Generation for Vocabulary Learning Using the Keyword Method [9.862827991755076]
  The 'keyword method' is an effective technique for learning vocabulary of a foreign language.<n>It involves creating a memorable visual link between what a word means and what its pronunciation in a foreign language sounds like.<n>We developed an application that combines the keyword method with text-to-image generators to externalise the memorable visual links into visuals.
  arXiv  Detail & Related papers  (2025-01-28T17:39:50Z)
• Language with Vision: a Study on Grounded Word and Sentence Embeddings [6.231247903840833]
  Grounding language in vision is an active field of research seeking to construct cognitively plausible word and sentence representations. The present study proposes a computational grounding model for pre-trained word embeddings. Our model effectively balances the interplay between language and vision by aligning textual embeddings with visual information.
  arXiv  Detail & Related papers  (2022-06-17T15:04:05Z)
• Comprehending and Ordering Semantics for Image Captioning [124.48670699658649]
  We propose a new recipe of Transformer-style structure, namely Comprehending and Ordering Semantics Networks (COS-Net) COS-Net unifies an enriched semantic comprehending and a learnable semantic ordering processes into a single architecture.
  arXiv  Detail & Related papers  (2022-06-14T15:51:14Z)
• Attention-Based Keyword Localisation in Speech using Visual Grounding [32.170748231414365]
  We investigate whether visually grounded speech models can also do keyword localisation. We show that attention provides a large gain in performance over previous visually grounded models. As in many other speech-image studies, we find that many of the incorrect localisations are due to semantic confusions.
  arXiv  Detail & Related papers  (2021-06-16T15:29:11Z)
• Accurate Word Representations with Universal Visual Guidance [55.71425503859685]
  This paper proposes a visual representation method to explicitly enhance conventional word embedding with multiple-aspect senses from visual guidance. We build a small-scale word-image dictionary from a multimodal seed dataset where each word corresponds to diverse related images. Experiments on 12 natural language understanding and machine translation tasks further verify the effectiveness and the generalization capability of the proposed approach.
  arXiv  Detail & Related papers  (2020-12-30T09:11:50Z)
• Contrastive Learning for Weakly Supervised Phrase Grounding [99.73968052506206]
  We show that phrase grounding can be learned by optimizing word-region attention. A key idea is to construct effective negative captions for learning through language model guided word substitutions. Our weakly supervised phrase grounding model trained on COCO-Captions shows a healthy gain of $5.7%$ to achieve $76.7%$ accuracy on Flickr30K benchmark.
  arXiv  Detail & Related papers  (2020-06-17T15:00:53Z)
• Learning to Recognise Words using Visually Grounded Speech [15.972015648122914]
  The model has been trained on pairs of images and spoken captions to create visually grounded embeddings. We investigate whether such a model can be used to recognise words by embedding isolated words and using them to retrieve images of their visual referents.
  arXiv  Detail & Related papers  (2020-05-31T12:48:37Z)
• On Vocabulary Reliance in Scene Text Recognition [79.21737876442253]
  Methods perform well on images with words within vocabulary but generalize poorly to images with words outside vocabulary. We call this phenomenon "vocabulary reliance" We propose a simple yet effective mutual learning strategy to allow models of two families to learn collaboratively.
  arXiv  Detail & Related papers  (2020-05-08T11:16:58Z)
• Visual Grounding in Video for Unsupervised Word Translation [91.47607488740647]
  We use visual grounding to improve unsupervised word mapping between languages. We learn embeddings from unpaired instructional videos narrated in the native language. We apply these methods to translate words from English to French, Korean, and Japanese.
  arXiv  Detail & Related papers  (2020-03-11T02:03:37Z)
• Learning Representations by Predicting Bags of Visual Words [55.332200948110895]
  Self-supervised representation learning targets to learn convnet-based image representations from unlabeled data. Inspired by the success of NLP methods in this area, in this work we propose a self-supervised approach based on spatially dense image descriptions.
  arXiv  Detail & Related papers  (2020-02-27T16:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.