Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026

• URL: http://arxiv.org/abs/2606.04730v1
• Date: Wed, 03 Jun 2026 11:13:37 GMT
• Title: Multilingual Long-Form Speech Instruction Following: KIT's Submission to IWSLT 2026
• Abstract summary: We present KIT's submission to the Long and Short Instruction Following tracks in the unconstrained setting.<n>Our approach combines a general data augmentation pipeline that converts short-form corpora into long-form training data.<n>We show that likelihood-based re-ranking, while highly effective for ASR, systematically degrades semantic tasks.
• Score: 57.48615919717867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the advent of Large Language Models, single-task and token-based multi-task models have evolved into instruction-based systems that infer task and target language implicitly from natural language prompts. This trend is reflected in IWSLT's Instruction Following Track, which this year introduced new tasks including an unknown surprise task, posing a genuine challenge against overfitting to known tasks. We present KIT's submission to the Long and Short Instruction Following tracks in the unconstrained setting. Our approach combines a general data augmentation pipeline that converts short-form corpora into long-form training data through segment concatenation, LLM-based label generation, and cross-lingual translation, yielding over 1M instances across six tasks and four languages. We further show that likelihood-based re-ranking, while highly effective for ASR, systematically degrades semantic tasks by spuriously selecting candidates generated from segmented audio processing rather than holistic long-form inference, a failure mode resolved by combining likelihood with Minimum Bayes Risk decoding.

Related papers

• Multi-Task Reinforcement Learning with Language-Encoded Gated Policy Networks [0.6345523830122167]
  Multi-task reinforcement learning often relies on task metadata to guide behavior across diverse objectives.<n>We present Lexical Policy Networks (LEXPOL), a language-conditioned mixture-of-policies architecture for multi-task RL.<n>LEXPOL encodes task metadata with a text encoder and uses a learned gating module to select or blend among sub-policies.
  arXiv  Detail & Related papers  (2025-10-07T17:12:24Z)
• KIT's Offline Speech Translation and Instruction Following Submission for IWSLT 2025 [56.61209412965054]
  We present the Karlsruhe Institute of Technology's submissions for the Offline ST and Instruction Following (IF) tracks.<n>We propose a pipeline that employs multiple automatic speech recognition systems, whose outputs are fused using an LLM with document-level context.<n>For the IF track, we develop an end-to-end model that integrates a speech encoder with an LLM to perform a wide range of instruction-following tasks.
  arXiv  Detail & Related papers  (2025-05-19T12:21:29Z)
• Few-Shot Multilingual Open-Domain QA from 5 Examples [44.04243892727856]
  We introduce a emphfew-shot learning approach to synthesise large-scale multilingual data from large language models (LLMs)<n>Our method begins with large-scale self-supervised pre-training using WikiData, followed by training on high-quality synthetic multilingual data generated by prompting LLMs with few-shot supervision.<n>The final model, textscFsModQA, significantly outperforms existing few-shot and supervised baselines in MLODQA and cross-lingual and monolingual retrieval.
  arXiv  Detail & Related papers  (2025-02-27T03:24:57Z)
• Contrastive Learning for Task-Independent SpeechLLM-Pretraining [14.531386555183596]
  Large language models (LLMs) excel in natural language processing.<n>Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs.<n>We propose a scalable, two-stage training approach.
  arXiv  Detail & Related papers  (2024-12-20T09:33:31Z)
• An End-to-End Speech Summarization Using Large Language Model [7.562198375754054]
  Speech Summarization (SSum) aims to generate human-like text summaries from spoken content. Research on large language models (LLMs) and multimodal information fusion has provided new insights. We propose an end-to-end SSum model that utilizes Q-Former as a connector for the audio-text modality.
  arXiv  Detail & Related papers  (2024-07-02T07:22:57Z)
• WavLLM: Towards Robust and Adaptive Speech Large Language Model [93.0773293897888]
  We introduce WavLLM, a robust and adaptive speech large language model with dual encoders, and a prompt-aware LoRA weight adapter. We validate the proposed model on universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also apply it to specialized datasets like Gaokao English listening comprehension set for SQA, and speech Chain-of-Thought (CoT) evaluation set.
  arXiv  Detail & Related papers  (2024-03-31T12:01:32Z)
• Fine-tuning Large Language Models with Sequential Instructions [2.546845645875049]
  We find that existing instruction-tuned models struggle to respond to queries with multiple instructions. We contend that part of the fine-tuning data mixture should be sequential--containing a chain of interrelated tasks. We automate this process by turning instructions in existing datasets into diverse and complex sequential instructions. Models that underwent our sequential instruction tuning show improved results in coding, maths, and open-ended generation.
  arXiv  Detail & Related papers  (2024-03-12T16:33:30Z)
• Meta-Task Prompting Elicits Embeddings from Large Language Models [54.757445048329735]
  We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation. We generate high-quality sentence embeddings from Large Language Models without the need for model fine-tuning. Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.
  arXiv  Detail & Related papers  (2024-02-28T16:35:52Z)
• Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models [54.58989938395976]
  We introduce a decomposed prompting approach for sequence labeling tasks.<n>We test our method on the Universal Dependencies part-of-speech tagging dataset for 38 languages.
  arXiv  Detail & Related papers  (2024-02-28T15:15:39Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)
• Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation [80.16548523140025]
  We extend the vanilla pretrain-finetune pipeline with extra code-switching restore task to bridge the gap between the pretrain and finetune stages. Our approach could narrow the cross-lingual sentence representation distance and improve low-frequency word translation with trivial computational cost.
  arXiv  Detail & Related papers  (2022-04-16T16:08:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.