Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding

• URL: http://arxiv.org/abs/2608.26925v1
• Date: Thu, 27 Aug 2026 10:22:07 GMT
• Title: Mapping Written Words to Spoken Words in a Different Language Using Only Visual Grounding
• Abstract summary: We consider how we can map a written English keyword to spoken realisations of that word in Hindi.<n>Our work demonstrates that cross-lingual word-to-speech mappings can be learned directly from visual grounding.
• Score: 23.312539472103566
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In many low-resource settings, even just eliciting speech for data collection is difficult. One promising approach has been to ask speakers to describe images. But how do we build models from such visually grounded speech data? Given a dataset of images with Hindi spoken captions, we consider how we can map a written English keyword to spoken realisations of that word in Hindi. Previous work trained end-to-end multimodal neural models. Instead, we explore a simpler alignment-based approach built on self-supervised speech representations. Written English tags are automatically obtained from images using off-the-shelf image captioning systems. Hindi utterances associated with the same keyword are then aligned (using self-supervised features), and alignment evidence is aggregated to identify recurring speech segments corresponding to the target word. Experiments evaluating keyword spotting and localization show that our alignment-based approach outperforms a previous attention-based neural model. We also show the benefit of incorporating negative examples during alignment. Our work demonstrates that cross-lingual word-to-speech mappings can be learned directly from visual grounding without transcriptions or explicit model training.

Related papers

• Connecting Speech to Words through Images [23.312539472103566]
  We present a visually grounded method for building a vocabulary of spoken words using only images and their spoken descriptions.<n>In spoken word retrieval and keyword spotting experiments, the proposed approach outperforms a strong neural baseline while being more interpretable.
  arXiv  Detail & Related papers  (2026-06-15T14:50:42Z)
• Layover or Direct Flight: Rethinking Audio-Guided Image Segmentation [65.7990140284317]
  We focus on object grounding, i.e., localizing an object of interest in a visual scene based on verbal human instructions.<n>To explore this possibility, we simplify the task by focusing on grounding from single-word spoken instructions.<n>Our results demonstrate that direct grounding from audio is not only feasible but, in some cases, even outperforms transcription-based methods.
  arXiv  Detail & Related papers  (2025-11-27T02:00:28Z)
• Translating speech with just images [23.104041372055466]
  We extend this connection by linking images to text via an existing image captioning system. This approach can be used for speech translation with just images by having the audio in a different language from the generated captions. We investigate such a system on a real low-resource language, Yorub'a, and propose a Yorub'a-to-English speech translation model.
  arXiv  Detail & Related papers  (2024-06-11T10:29:24Z)
• Pixel Aligned Language Models [94.32841818609914]
  We develop a vision-language model that can take locations as either inputs or outputs. When taking locations as inputs, the model performs location-conditioned captioning, which generates captions for the indicated object or region. Our model is pre-trained on the Localized Narrative dataset, which contains pixel-word-aligned captioning from human attention.
  arXiv  Detail & Related papers  (2023-12-14T18:57:58Z)
• Visually grounded few-shot word learning in low-resource settings [23.826000011632917]
  We propose a visual grounded speech model that learns new words and their visual depictions from just a few word-image example pairs. Our approach involves using the given word-image example pairs to mine new unsupervised word-image training pairs from large collections of unlabelled speech and images. With this new model, we achieve better performance with fewer shots than previous approaches on an existing English benchmark.
  arXiv  Detail & Related papers  (2023-06-20T08:27:42Z)
• Towards visually prompted keyword localisation for zero-resource spoken languages [27.696096343873215]
  We formalise the task of visually prompted keyword localisation (VPKL) VPKL is given an image of a keyword, detect and predict where in an utterance the keyword occurs. We show that these innovations give improvements in VPKL over an existing speech-vision model.
  arXiv  Detail & Related papers  (2022-10-12T14:17:34Z)
• Self-Supervised Speech Representation Learning: A Review [105.1545308184483]
  Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. This review presents approaches for self-supervised speech representation learning and their connection to other research areas.
  arXiv  Detail & Related papers  (2022-05-21T16:52:57Z)
• Attention-Based Keyword Localisation in Speech using Visual Grounding [32.170748231414365]
  We investigate whether visually grounded speech models can also do keyword localisation. We show that attention provides a large gain in performance over previous visually grounded models. As in many other speech-image studies, we find that many of the incorrect localisations are due to semantic confusions.
  arXiv  Detail & Related papers  (2021-06-16T15:29:11Z)
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision [110.66085917826648]
  We develop a technique that extrapolates multimodal alignments to language-only data by contextually mapping language tokens to their related images. "vokenization" is trained on relatively small image captioning datasets and we then apply it to generate vokens for large language corpora. Trained with these contextually generated vokens, our visually-supervised language models show consistent improvements over self-supervised alternatives on multiple pure-language tasks.
  arXiv  Detail & Related papers  (2020-10-14T02:11:51Z)
• Seeing wake words: Audio-visual Keyword Spotting [103.12655603634337]
  KWS-Net is a novel convolutional architecture that uses a similarity map intermediate representation to separate the task into sequence matching and pattern detection. We show that our method generalises to other languages, specifically French and German, and achieves a comparable performance to English with less language specific data.
  arXiv  Detail & Related papers  (2020-09-02T17:57:38Z)
• Visual Grounding in Video for Unsupervised Word Translation [91.47607488740647]
  We use visual grounding to improve unsupervised word mapping between languages. We learn embeddings from unpaired instructional videos narrated in the native language. We apply these methods to translate words from English to French, Korean, and Japanese.
  arXiv  Detail & Related papers  (2020-03-11T02:03:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.