LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams

• URL: http://arxiv.org/abs/2504.17366v1
• Date: Thu, 24 Apr 2025 08:27:48 GMT
• Title: LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams
• Authors: Yongxuan Wu, Runyu Chen, Peiyu Liu, Hongjin Qian,
• Abstract summary: We construct the first spoken long-text dataset, derived from live streams, to reflect the redundancy-rich and conversational nature of real-world scenarios.<n>We evaluate both popular LLMs and specialized methods to assess their ability to understand long-contexts in these tasks.<n>Our findings highlight key limitations of current methods and suggest future directions for improving long-context understanding.
• Score: 4.917265821383127
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-context understanding poses significant challenges in natural language processing, particularly for real-world dialogues characterized by speech-based elements, high redundancy, and uneven information density. Although large language models (LLMs) achieve impressive results on existing benchmarks, these datasets fail to reflect the complexities of such texts, limiting their applicability to practical scenarios. To bridge this gap, we construct the first spoken long-text dataset, derived from live streams, designed to reflect the redundancy-rich and conversational nature of real-world scenarios. We construct tasks in three categories: retrieval-dependent, reasoning-dependent, and hybrid. We then evaluate both popular LLMs and specialized methods to assess their ability to understand long-contexts in these tasks. Our results show that current methods exhibit strong task-specific preferences and perform poorly on highly redundant inputs, with no single method consistently outperforming others. We propose a new baseline that better handles redundancy in spoken text and achieves strong performance across tasks. Our findings highlight key limitations of current methods and suggest future directions for improving long-context understanding. Finally, our benchmark fills a gap in evaluating long-context spoken language understanding and provides a practical foundation for developing real-world e-commerce systems. The code and benchmark are available at https://github.com/Yarayx/livelongbench.

Related papers

• A Survey on Transformer Context Extension: Approaches and Evaluation [12.727256455697011]
  Large language models (LLMs) based on Transformer have been widely applied in the filed of natural language processing (NLP)<n>However, when it comes to long context scenarios, the performance of LLMs degrades due to some challenges.
  arXiv  Detail & Related papers  (2025-03-17T15:44:09Z)
• WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale [86.25450054683172]
  WildLong extracts meta-information from real user queries to produce scalable data.<n>It supports multi-document reasoning, such as cross-document comparison and aggregation.<n>It surpasses existing open-source long-context-optimized models across benchmarks.
  arXiv  Detail & Related papers  (2025-02-23T18:59:09Z)
• Generalizing From Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning [103.65680870130839]
  We investigate how to design instruction data for the post-training phase of a long context pre-trained model. Our controlled study reveals that models instruction-tuned on short contexts can effectively generalize to longer ones. Based on these findings, we propose context synthesis, a novel data synthesis framework.
  arXiv  Detail & Related papers  (2025-02-21T17:02:40Z)
• Emulating Retrieval Augmented Generation via Prompt Engineering for Enhanced Long Context Comprehension in LLMs [23.960451986662996]
  This paper proposes a method that emulates Retrieval Augmented Generation (RAG) through specialized prompt engineering and chain-of-thought reasoning.<n>We evaluate our approach on selected tasks from BABILong, which interleaves standard bAbI QA problems with large amounts of distractor text.
  arXiv  Detail & Related papers  (2025-02-18T02:49:40Z)
• A Controlled Study on Long Context Extension and Generalization in LLMs [85.4758128256142]
  Broad textual understanding and in-context learning require language models that utilize full document contexts. Due to the implementation challenges associated with directly training long-context models, many methods have been proposed for extending models to handle long contexts. We implement a controlled protocol for extension methods with a standardized evaluation, utilizing consistent base models and extension data.
  arXiv  Detail & Related papers  (2024-09-18T17:53:17Z)
• NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context Window? [37.64593022203498]
  NeedleBench is a framework consisting of progressively more challenging tasks for assessing bilingual long-context capabilities. We use the framework to assess how well the leading open-source models can identify key information relevant to the question. We propose the Ancestral Trace Challenge to mimic the complexity of logical reasoning challenges that are likely to be present in real-world long-context tasks.
  arXiv  Detail & Related papers  (2024-07-16T17:59:06Z)
• KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches [52.02764371205856]
  Long context capability is a crucial competency for large language models (LLMs) This work provides a taxonomy of current methods and evaluating 10+ state-of-the-art approaches across seven categories of long context tasks.
  arXiv  Detail & Related papers  (2024-07-01T17:59:47Z)
• XL$^2$Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies [45.31042312867939]
  Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks but are constrained by their small context window sizes. Various efforts have been proposed to expand the context window to accommodate even up to 200K input tokens. We introduce a benchmark for extremely long context understanding with long-range dependencies, XL$2$Bench.
  arXiv  Detail & Related papers  (2024-04-08T12:29:07Z)
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding [58.20031627237889]
  LongBench is the first bilingual, multi-task benchmark for long context understanding. It comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese)
  arXiv  Detail & Related papers  (2023-08-28T11:53:40Z)
• SCROLLS: Standardized CompaRison Over Long Language Sequences [62.574959194373264]
  We introduce SCROLLS, a suite of tasks that require reasoning over long texts. SCROLLS contains summarization, question answering, and natural language inference tasks. We make all datasets available in a unified text-to-text format and host a live leaderboard to facilitate research on model architecture and pretraining methods.
  arXiv  Detail & Related papers  (2022-01-10T18:47:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.