Boosting Large Language Model for Speech Synthesis: An Empirical Study
- URL: http://arxiv.org/abs/2401.00246v1
- Date: Sat, 30 Dec 2023 14:20:04 GMT
- Title: Boosting Large Language Model for Speech Synthesis: An Empirical Study
- Authors: Hongkun Hao, Long Zhou, Shujie Liu, Jinyu Li, Shujie Hu, Rui Wang,
Furu Wei
- Abstract summary: Large language models (LLMs) have made significant advancements in natural language processing and are concurrently extending the language ability to other modalities, such as speech and vision.
We conduct a comprehensive empirical exploration of boosting LLMs with the ability to generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech synthesis model VALL-E.
We compare three integration methods between LLMs and speech models, including directly fine-tuned LLMs, superposed layers of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text encoder
- Score: 86.89548753080432
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Large language models (LLMs) have made significant advancements in natural
language processing and are concurrently extending the language ability to
other modalities, such as speech and vision. Nevertheless, most of the previous
work focuses on prompting LLMs with perception abilities like auditory
comprehension, and the effective approach for augmenting LLMs with speech
synthesis capabilities remains ambiguous. In this paper, we conduct a
comprehensive empirical exploration of boosting LLMs with the ability to
generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech
synthesis model VALL-E. We compare three integration methods between LLMs and
speech synthesis models, including directly fine-tuned LLMs, superposed layers
of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text
encoder. Experimental results show that, using LoRA method to fine-tune LLMs
directly to boost the speech synthesis capability does not work well, and
superposed LLMs and VALL-E can improve the quality of generated speech both in
speaker similarity and word error rate (WER). Among these three methods,
coupled methods leveraging LLMs as the text encoder can achieve the best
performance, making it outperform original speech synthesis models with a
consistently better speaker similarity and a significant (10.9%) WER reduction.
Related papers
- LLM2CLIP: Powerful Language Model Unlocks Richer Visual Representation [60.02145113467427]
This work introduces a fine-tuning approach that integrates large language models with the pretrained CLIP visual encoder.
To address the challenge of LLMs' autoregressive nature, we propose a caption-to-caption contrastive learning framework.
Our method achieves substantial performance gains on various downstream tasks.
arXiv Detail & Related papers (2024-11-07T18:59:16Z) - LLM Gesticulator: Leveraging Large Language Models for Scalable and Controllable Co-Speech Gesture Synthesis [4.762487293009696]
We present LLM Gesticulator, an audio-driven co-speech gesture generation framework.
Our framework synthesizes full-body animations that are rhythmically aligned with the input audio while exhibiting natural movements and editability.
Our method also exhibits strong controllability where the content, style of the generated gestures can be controlled by text prompt.
arXiv Detail & Related papers (2024-10-06T12:53:07Z) - Enhancing Multilingual Speech Generation and Recognition Abilities in LLMs with Constructed Code-switched Data [30.966072545451183]
We propose a MutltiLingual MultiTask (MLMT) model, integrating multilingual speech generation and recognition tasks within the single LLM.
We develop an effective data construction approach that splits and equips words from different languages to equip synthesiss with CS ability without relying on CS data.
arXiv Detail & Related papers (2024-09-17T08:11:07Z) - Prompting Large Language Models with Audio for General-Purpose Speech Summarization [13.415189715216354]
We introduce a framework for speech summarization that leverages the processing and reasoning capabilities of large language models (LLMs)
We propose an end-to-end system that combines an instruction-tuned LLM with an audio encoder that converts speech into token representations that the LLM can interpret.
arXiv Detail & Related papers (2024-06-10T02:04:28Z) - Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizing [56.75702900542643]
We introduce AlphaLLM for the self-improvements of Large Language Models.
It integrates Monte Carlo Tree Search (MCTS) with LLMs to establish a self-improving loop.
Our experimental results show that AlphaLLM significantly enhances the performance of LLMs without additional annotations.
arXiv Detail & Related papers (2024-04-18T15:21:34Z) - ST-LLM: Large Language Models Are Effective Temporal Learners [58.79456373423189]
Large Language Models (LLMs) have showcased impressive capabilities in text comprehension and generation.
How to effectively encode and understand videos in video-based dialogue systems remains to be solved.
We propose ST-LLM, an effective video-LLM baseline with spatial-temporal sequence modeling inside LLM.
arXiv Detail & Related papers (2024-03-30T10:11:26Z) - An Embarrassingly Simple Approach for LLM with Strong ASR Capacity [56.30595787061546]
We focus on solving one of the most important tasks in the field of speech processing, with speech foundation encoders and large language models (LLM)
Recent works have complex designs such as compressing the output temporally for the speech encoder, tackling modal alignment for the projector, and utilizing parameter-efficient fine-tuning for the LLM.
We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task.
arXiv Detail & Related papers (2024-02-13T23:25:04Z) - Speech Translation with Large Language Models: An Industrial Practice [64.5419534101104]
We introduce LLM-ST, a novel and effective speech translation model constructed upon a pre-trained large language model (LLM)
By integrating the large language model (LLM) with a speech encoder and employing multi-task instruction tuning, LLM-ST can produce accurate timestamped transcriptions and translations.
Through rigorous experimentation on English and Chinese datasets, we showcase the exceptional performance of LLM-ST.
arXiv Detail & Related papers (2023-12-21T05:32:49Z) - Prompting Large Language Models with Speech Recognition Abilities [31.77576008965215]
We extend the capabilities of large language models by directly attaching a small audio encoder allowing it to perform speech recognition.
Experiments on MultilingualSpeech show that incorporating a conformer encoder into the open sourced LLaMA-7B allows it to outperform monolingual baselines by 18%.
arXiv Detail & Related papers (2023-07-21T08:39:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.