NAVER LABS Europe Submission to the Instruction-following Track

• URL: http://arxiv.org/abs/2506.01808v1
• Date: Mon, 02 Jun 2025 15:52:57 GMT
• Title: NAVER LABS Europe Submission to the Instruction-following Track
• Abstract summary: We describe NAVER LABS Europe submission to the instruction-following speech processing short track at IWSLT 2025.<n>We develop systems that can simultaneously perform ASR, ST, and SQA tasks from English speech input into the following target languages: Chinese, Italian, and German.
• Score: 25.9221252675616
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In this paper we describe NAVER LABS Europe submission to the instruction-following speech processing short track at IWSLT 2025. We participate in the constrained settings, developing systems that can simultaneously perform ASR, ST, and SQA tasks from English speech input into the following target languages: Chinese, Italian, and German. Our solution leverages two pretrained modules: (1) a speech-to-LLM embedding projector trained using representations from the SeamlessM4T-v2-large speech encoder; and (2) LoRA adapters trained on text data on top of a Llama-3.1-8B-Instruct. These modules are jointly loaded and further instruction-tuned for 1K steps on multilingual and multimodal data to form our final system submitted for evaluation.

Related papers

• NAVER LABS Europe Submission to the Instruction-following 2026 Short Track [12.471740115082126]
  We describe NAVER LABS Europe's submission to the instruction-following speech processing short track at IWSLT 2026.<n>We develop systems capable of jointly performing ASR, ST, and SQA from English speech into Chinese, Italian, and German.<n>This year's results place our system tied for first place in the overall short track ranking.
  arXiv  Detail & Related papers  (2026-07-02T09:52:08Z)
• Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision [36.3996368188181]
  Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction.<n>Recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector.<n>We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector.
  arXiv  Detail & Related papers  (2026-03-07T04:09:47Z)
• AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning [49.68129589035101]
  We introduce AZeroS (Auden Zero-instruction-tuned Speech-LLM), which is trained on speech-text pairs derived from publicly available corpora.<n>AZeroS achieves state-of-the-art performance on both semantic and paralinguistic benchmarks.
  arXiv  Detail & Related papers  (2025-12-31T04:05:04Z)
• SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challenge [3.9836024799656053]
  Our system integrates a parallel-speech-encoder architecture with a large language model (LLM) to form a unified multilingual ASR framework.<n>The SHNU-mASR system achieves an overall character/word error rate (CER/WER) of 11.76% on the blind evaluation set of the INTERSPEECH 2025 MLC-SLM Challenge.
  arXiv  Detail & Related papers  (2025-07-04T07:10:33Z)
• KIT's Offline Speech Translation and Instruction Following Submission for IWSLT 2025 [56.61209412965054]
  We present the Karlsruhe Institute of Technology's submissions for the Offline ST and Instruction Following (IF) tracks.<n>We propose a pipeline that employs multiple automatic speech recognition systems, whose outputs are fused using an LLM with document-level context.<n>For the IF track, we develop an end-to-end model that integrates a speech encoder with an LLM to perform a wide range of instruction-following tasks.
  arXiv  Detail & Related papers  (2025-05-19T12:21:29Z)
• Zero-resource Speech Translation and Recognition with LLMs [38.11535502039386]
  We propose to leverage a multilingual Large Language Model (LLM) to perform ST and ASR in languages for which the model has never seen paired audio-text data.<n>We achieve this by using a pre-trained multilingual speech encoder, a multilingual LLM, and a lightweight adaptation module that maps the audio representations to the token embedding space of the LLM.
  arXiv  Detail & Related papers  (2024-12-24T17:37:11Z)
• CMU's IWSLT 2024 Simultaneous Speech Translation System [80.15755988907506]
  This paper describes CMU's submission to the IWSLT 2024 Simultaneous Speech Translation (SST) task for translating English speech to German text in a streaming manner. Our end-to-end speech-to-text (ST) system integrates the WavLM speech encoder, a modality adapter, and the Llama2-7B-Base model as the decoder.
  arXiv  Detail & Related papers  (2024-08-14T10:44:51Z)
• NAIST Simultaneous Speech Translation System for IWSLT 2024 [18.77311658086372]
  This paper describes NAIST's submission to the simultaneous track of the IWSLT 2024 Evaluation Campaign. We develop a multilingual end-to-end speech-to-text translation model combining two pre-trained language models, HuBERT and mBART. We trained this model with two decoding policies, Local Agreement (LA) and AlignAtt. Our speech-to-speech translation method is a cascade of the above speech-to-text model and an incremental text-to-speech (TTS) module.
  arXiv  Detail & Related papers  (2024-06-30T20:41:02Z)
• ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation [79.66359274050885]
  We present ComSL, a speech-language model built atop a composite architecture of public pretrained speech-only and language-only models. Our approach has demonstrated effectiveness in end-to-end speech-to-text translation tasks.
  arXiv  Detail & Related papers  (2023-05-24T07:42:15Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)
• VANI: Very-lightweight Accent-controllable TTS for Native and Non-native speakers with Identity Preservation [25.210244564579522]
  We introduce VANI, a very lightweight multi-lingual accent controllable speech synthesis system. We utilize the Indic languages dataset, released for LIMMITS 2023 as part of ICASSP Signal Processing Grand Challenge, to synthesize speech in 3 different languages.
  arXiv  Detail & Related papers  (2023-03-14T01:55:41Z)
• Textless Speech-to-Speech Translation on Real Data [49.134208897722246]
  We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language. We tackle the challenge in modeling multi-speaker target speech and train the systems with real-world S2ST data.
  arXiv  Detail & Related papers  (2021-12-15T18:56:35Z)
• SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding [61.02342238771685]
  Spoken language understanding requires a model to analyze input acoustic signal to understand its linguistic content and make predictions. Various pre-training methods have been proposed to learn rich representations from large-scale unannotated speech and text. We propose a novel semi-supervised learning framework, SPLAT, to jointly pre-train the speech and language modules.
  arXiv  Detail & Related papers  (2020-10-05T19:29:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.