SeaLLMs-Audio: Large Audio-Language Models for Southeast Asia

• URL: http://arxiv.org/abs/2511.01670v1
• Date: Mon, 03 Nov 2025 15:32:58 GMT
• Title: SeaLLMs-Audio: Large Audio-Language Models for Southeast Asia
• Abstract summary: We introduce SeaLLMs-Audio, the first large audio-language model (LALM) tailored for multiple Southeast Asian languages.<n>SeaLLMs-Audio exhibits strong performance across diverse audio-centric tasks, spanning fine-grained audio understanding and voice-based interaction.<n>It supports a wide range of tasks, including audio analysis tasks such as Audio Captioning, Automatic Speech Recognition, Speech-to-Text Translation, Speech Emotion Recognition, Speech Question Answering, and Speech Summarization.
• Score: 40.53123362174684
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce SeaLLMs-Audio, the first large audio-language model (LALM) tailored for multiple Southeast Asian (SEA) languages-Indonesian (id), Thai (th), and Vietnamese (vi)-alongside English (en) and Chinese (zh). Trained on a large-scale audio corpus, SeaLLMs-Audio exhibits strong performance across diverse audio-centric tasks, spanning fine-grained audio understanding and voice-based interaction. Its key features include: 1) Multilingual: the model primarily supports 5 languages, namely Indonesian, Thai, Vietnamese, English, and Chinese; 2) Multimodal: the model accepts flexible input modalities, including audio only, text only, as well as audio with text; 3) Multi-task: the model supports a wide range of tasks, including audio analysis tasks such as Audio Captioning, Automatic Speech Recognition, Speech-to-Text Translation, Speech Emotion Recognition, Speech Question Answering, and Speech Summarization. It also enables voice-based dialogue, including answering factual, mathematical, and general knowledge queries. As a significant step towards advancing audio LLMs in Southeast Asia, we expect SeaLLMs-Audio to benefit both the regional research community and industry. To automate LALM evaluation for Southeast Asia, we introduce SeaBench-Audio, a benchmark spanning multiple tasks. Experiments show that SeaLLMs-Audio achieves competitive performance compared with other LALMs on SEA languages.

Related papers

• SEA-SpeechBench: A Large-Scale Multitask Benchmark for Speech Understanding Across Southeast Asia [20.33019395334099]
  We introduce SEA-SpeechBench, the first large-scale benchmark that evaluates speech understanding in 11 Southeast Asian languages.<n>Our benchmark comprises 9 diverse tasks across 3 categories: speech processing (automatic speech recognition, speech translation, spoken question answering), paralinguistic analysis (emotion, gender, age, speaker recognition), and temporal understanding.<n>We implement multilingual prompting in both native SEA languages and English to reflect user interactions with audio-language models.
  arXiv  Detail & Related papers  (2026-09-09T03:39:18Z)
• Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision [36.3996368188181]
  Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction.<n>Recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector.<n>We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector.
  arXiv  Detail & Related papers  (2026-03-07T04:09:47Z)
• UALM: Unified Audio Language Model for Understanding, Generation and Reasoning [124.19449187588832]
  Unified Audio Language Model (UALM) aims to unify audio understanding, text-to-audio generation, and multimodal reasoning in a single model.<n>We first present UALM-Gen, a text-to-audio language model that directly predicts audio tokens and is comparable to state-of-the-art diffusion-based models.<n>We present UALM-Reason, a multimodal reasoning model that utilizes both text and audio in the intermediate thinking steps to facilitate complex generation tasks.
  arXiv  Detail & Related papers  (2025-10-13T22:55:01Z)
• Step-Audio 2 Technical Report [120.58375054866815]
  Step-Audio 2 is an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation.<n>By integrating a latent audio encoder and reasoning-centric reinforcement learning (RL), Step-Audio 2 achieves promising performance in automatic speech recognition (ASR) and audio understanding.
  arXiv  Detail & Related papers  (2025-07-22T14:23:55Z)
• Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context [45.56363286769136]
  We introduce Solla, a novel framework designed to understand speech-based questions and hear the acoustic context concurrently.<n>Solla incorporates an audio tagging module to effectively identify and represent audio events, as well as an ASR-assisted prediction method to improve comprehension of spoken content.<n>We propose a new benchmark dataset called SA-Eval, which includes three tasks: audio event classification, audio captioning, and audio question answering.
  arXiv  Detail & Related papers  (2025-03-19T15:34:21Z)
• Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models [58.43486430996411]
  Large Audio-Language Models (LALMs) have recently unlocked audio dialogue capabilities, enabling direct spoken exchanges with humans.<n>We propose an Audio Dialogue Understanding Benchmark (ADU-Bench) to evaluate the performance of LALMs in the open-ended audio dialogue understanding.<n>ADU-Bench includes over 20,000 open-ended audio dialogues for the assessment of LALMs.
  arXiv  Detail & Related papers  (2024-12-06T16:34:15Z)
• Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models [13.855545744177586]
  This paper evaluates audio language models on Thai, a low-resource language, and finds that they lack emergent cross-lingual abilities despite their multilingual foundations.<n>Our experiments provide insights into improving instruction-following in low-resource languages by balancing language-specific and multilingual training data.<n>The proposed model, Typhoon-Audio, significantly outperforms existing open-source models and achieves performance comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai.
  arXiv  Detail & Related papers  (2024-09-17T09:04:03Z)
• Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities [37.02115473120654]
  Augmenting large language models (LLMs) to understand audio is critically important for diverse real-world applications. In this paper, we propose Audio Flamingo, a novel audio language model with 1) strong audio understanding abilities, 2) the ability to quickly adapt to unseen tasks via in-context learning and retrieval, and 3) strong multi-turn dialogue abilities.
  arXiv  Detail & Related papers  (2024-02-02T18:58:34Z)
• LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT [65.69648099999439]
  Generative Pre-trained Transformer (GPT) models have achieved remarkable performance on various natural language processing tasks. We propose LauraGPT, a novel unified audio-and-text GPT-based LLM for audio recognition, understanding, and generation.
  arXiv  Detail & Related papers  (2023-10-07T03:17:59Z)
• AudioPaLM: A Large Language Model That Can Speak and Listen [79.44757696533709]
  We introduce AudioPaLM, a large language model for speech understanding and generation. AudioPaLM fuses text-based and speech-based language models. It can process and generate text and speech with applications including speech recognition and speech-to-speech translation.
  arXiv  Detail & Related papers  (2023-06-22T14:37:54Z)
• AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head [82.69233563811487]
  Large language models (LLMs) have exhibited remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. We propose a multi-modal AI system named AudioGPT, which complements LLMs with foundation models to process complex audio information.
  arXiv  Detail & Related papers  (2023-04-25T17:05:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.