VILAS: Exploring the Effects of Vision and Language Context in Automatic
Speech Recognition
- URL: http://arxiv.org/abs/2305.19972v2
- Date: Mon, 18 Dec 2023 12:29:00 GMT
- Title: VILAS: Exploring the Effects of Vision and Language Context in Automatic
Speech Recognition
- Authors: Ziyi Ni and Minglun Han and Feilong Chen and Linghui Meng and Jing Shi
and Pin Lv and Bo Xu
- Abstract summary: ViLaS (Vision and Language into Automatic Speech Recognition) is a novel multimodal ASR model based on the continuous integrate-and-fire (CIF) mechanism.
To explore the effects of integrating vision and language, we create VSDial, a multimodal ASR dataset with multimodal context cues in both Chinese and English versions.
- Score: 18.19998336526969
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Enhancing automatic speech recognition (ASR) performance by leveraging
additional multimodal information has shown promising results in previous
studies. However, most of these works have primarily focused on utilizing
visual cues derived from human lip motions. In fact, context-dependent visual
and linguistic cues can also benefit in many scenarios. In this paper, we first
propose ViLaS (Vision and Language into Automatic Speech Recognition), a novel
multimodal ASR model based on the continuous integrate-and-fire (CIF)
mechanism, which can integrate visual and textual context simultaneously or
separately, to facilitate speech recognition. Next, we introduce an effective
training strategy that improves performance in modal-incomplete test scenarios.
Then, to explore the effects of integrating vision and language, we create
VSDial, a multimodal ASR dataset with multimodal context cues in both Chinese
and English versions. Finally, empirical results are reported on the public
Flickr8K and self-constructed VSDial datasets. We explore various cross-modal
fusion schemes, analyze fine-grained crossmodal alignment on VSDial, and
provide insights into the effects of integrating multimodal information on
speech recognition.
Related papers
- Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model [3.3772986620114387]
We introduce ISA-HOI, which extensively leverages knowledge from CLIP, aligning interactive semantics between visual and textual features.
Our method achieves competitive results on the HICO-DET and V-COCO benchmarks with much fewer training epochs, and outperforms the state-of-the-art under zero-shot settings.
arXiv Detail & Related papers (2024-04-19T07:24:32Z) - Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation [55.15299351110525]
This paper explores sentence-level multilingual Visual Speech Recognition (VSR) that can recognize different languages with a single trained model.
We propose a novel training strategy, processing with visual speech units.
We set new state-of-the-art multilingual VSR performances by achieving comparable performances to the previous language-specific VSR models.
arXiv Detail & Related papers (2024-01-18T08:46:02Z) - Towards More Unified In-context Visual Understanding [74.55332581979292]
We present a new ICL framework for visual understanding with multi-modal output enabled.
First, we quantize and embed both text and visual prompt into a unified representational space.
Then a decoder-only sparse transformer architecture is employed to perform generative modeling on them.
arXiv Detail & Related papers (2023-12-05T06:02:21Z) - Contextual Object Detection with Multimodal Large Language Models [78.30374204127418]
We introduce a novel research problem of contextual object detection.
Three representative scenarios are investigated, including the language cloze test, visual captioning, and question answering.
We present ContextDET, a unified multimodal model that is capable of end-to-end differentiable modeling of visual-language contexts.
arXiv Detail & Related papers (2023-05-29T17:50:33Z) - Cross-Modal Global Interaction and Local Alignment for Audio-Visual
Speech Recognition [21.477900473255264]
We propose a cross-modal global interaction and local alignment (GILA) approach for audio-visual speech recognition (AVSR)
Specifically, we design a global interaction model to capture the A-V complementary relationship on modality level, as well as a local alignment approach to model the A-V temporal consistency on frame level.
Our GILA outperforms the supervised learning state-of-the-art on public benchmarks LRS3 and LRS2.
arXiv Detail & Related papers (2023-05-16T06:41:25Z) - Accommodating Audio Modality in CLIP for Multimodal Processing [48.83906067348211]
We extend the Vision-Language model CLIP to accommodate the audio modality for Vision-Language-Audio multimodal processing.
Specifically, we apply inter-modal and intra-modal contrastive learning to explore the correlation between audio and other modalities.
Our proposed CLIP4VLA model is validated in different downstream tasks including video retrieval and video captioning.
arXiv Detail & Related papers (2023-03-12T06:57:01Z) - VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for
Speech Representation Learning [119.49605266839053]
We propose a unified cross-modal representation learning framework VATLM (Visual-Audio-Text Language Model)
The proposed VATLM employs a unified backbone network to model the modality-independent information.
In order to integrate these three modalities into one shared semantic space, VATLM is optimized with a masked prediction task of unified tokens.
arXiv Detail & Related papers (2022-11-21T09:10:10Z) - An Exploration of Self-Supervised Pretrained Representations for
End-to-End Speech Recognition [98.70304981174748]
We focus on the general applications of pretrained speech representations, on advanced end-to-end automatic speech recognition (E2E-ASR) models.
We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
arXiv Detail & Related papers (2021-10-09T15:06:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.