LLM$\times$MapReduce-V2: Entropy-Driven Convolutional Test-Time Scaling for Generating Long-Form Articles from Extremely Long Resources

• URL: http://arxiv.org/abs/2504.05732v2
• Date: Tue, 15 Apr 2025 03:28:58 GMT
• Title: LLM$\times$MapReduce-V2: Entropy-Driven Convolutional Test-Time Scaling for Generating Long-Form Articles from Extremely Long Resources
• Authors: Haoyu Wang, Yujia Fu, Zhu Zhang, Shuo Wang, Zirui Ren, Xiaorong Wang, Zhili Li, Chaoqun He, Bo An, Zhiyuan Liu, Maosong Sun,
• Abstract summary: Long-form generation is crucial for a wide range of practical applications.<n>While short-to-long generations have received considerable attention, generating long texts from extremely long resources remains relatively underexplored.<n>We propose LLM$times$MapReduce-V2, a novel test-time scaling strategy designed to enhance the ability of large language models to process extremely long inputs.
• Score: 65.41986915457058
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-form generation is crucial for a wide range of practical applications, typically categorized into short-to-long and long-to-long generation. While short-to-long generations have received considerable attention, generating long texts from extremely long resources remains relatively underexplored. The primary challenge in long-to-long generation lies in effectively integrating and analyzing relevant information from extensive inputs, which remains difficult for current large language models (LLMs). In this paper, we propose LLM$\times$MapReduce-V2, a novel test-time scaling strategy designed to enhance the ability of LLMs to process extremely long inputs. Drawing inspiration from convolutional neural networks, which iteratively integrate local features into higher-level global representations, LLM$\times$MapReduce-V2 utilizes stacked convolutional scaling layers to progressively expand the understanding of input materials. Both quantitative and qualitative experimental results demonstrate that our approach substantially enhances the ability of LLMs to process long inputs and generate coherent, informative long-form articles, outperforming several representative baselines. Both LLM$\times$MapReduce-V2 and SurveyEval are publicly available at https://github.com/thunlp/LLMxMapReduce .

Related papers

• Shifting Long-Context LLMs Research from Input to Output [32.227507695283144]
  This paper advocates for a paradigm shift in NLP research toward addressing the challenges of long-form generation.<n> Tasks such as novel writing, long-term planning, and complex reasoning require models to understand extensive contexts and produce coherent, contextually rich, and logically consistent extended text.<n>We underscore the importance of this under-explored domain and call for focused efforts to develop foundational LLMs tailored for generating high-quality, long-form outputs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:37Z)
• LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm [21.661578831520963]
  Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks.<n>Our analysis reveals that current LLMs struggle with length requirements and information density in long-text generation.<n>We present LongEval, a benchmark that evaluates long-text generation through both direct and plan-based generation paradigms.
  arXiv  Detail & Related papers  (2025-02-26T12:46:36Z)
• Language Models can Self-Lengthen to Generate Long Texts [74.96074422345806]
  This paper introduces an innovative iterative training framework called Self-Lengthen. It leverages only the intrinsic knowledge and skills of Large Language Models without the need for auxiliary data or proprietary models. Experiments on benchmarks and human evaluations show that Self-Lengthen outperforms existing methods in long-text generation.
  arXiv  Detail & Related papers  (2024-10-31T13:47:10Z)
• LLM$\times$MapReduce: Simplified Long-Sequence Processing using Large Language Models [73.13933847198395]
  We propose a training-free framework for processing long texts, utilizing a divide-and-conquer strategy to achieve comprehensive document understanding. The proposed LLM$times$MapReduce framework splits the entire document into several chunks for LLMs to read and then aggregates the intermediate answers to produce the final output.
  arXiv  Detail & Related papers  (2024-10-12T03:13:44Z)
• Scaling Up Summarization: Leveraging Large Language Models for Long Text Extractive Summarization [0.27624021966289597]
  This paper introduces EYEGLAXS, a framework that leverages Large Language Models (LLMs) for extractive summarization. EYEGLAXS focuses on extractive summarization to ensure factual and grammatical integrity. The system sets new performance benchmarks on well-known datasets like PubMed and ArXiv.
  arXiv  Detail & Related papers  (2024-08-28T13:52:19Z)
• Long Context Alignment with Short Instructions and Synthesized Positions [56.1267385315404]
  This paper introduces Step-Skipping Alignment (SkipAlign) It is a new technique designed to enhance the long-context capabilities of Large Language Models (LLMs) With a careful selection of the base model and alignment datasets, SkipAlign with only 6B parameters achieves it's best performance and comparable with strong baselines like GPT-3.5-Turbo-16K on LongBench.
  arXiv  Detail & Related papers  (2024-05-07T01:56:22Z)
• Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding [78.36702055076456]
  This paper introduces Multi-scale Positional. (Ms-PoE) which is a simple yet effective plug-and-play approach to enhance the capacity of. LLMs to handle relevant information located in the middle of the context.
  arXiv  Detail & Related papers  (2024-03-05T04:58:37Z)
• TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documents [34.52684986240312]
  We introduce TextGenSHAP, an efficient post-hoc explanation method incorporating LM-specific techniques. We demonstrate that this leads to significant increases in speed compared to conventional Shapley value computations. In addition, we demonstrate how real-time Shapley values can be utilized in two important scenarios.
  arXiv  Detail & Related papers  (2023-12-03T04:35:04Z)
• LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models [83.98062659664785]
  Large language models (LLMs) typically train on short text segments (e.g., 4K tokens) due to the quadratic complexity of their Transformer architectures. This work identifies three major factors contributing to this length generalization failure. We propose LM-Infinite, a simple and effective method for enhancing LLMs' capabilities of handling long contexts.
  arXiv  Detail & Related papers  (2023-08-30T16:47:51Z)
• Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs [59.74814230246034]
  Large Language Models (LLMs) have been proven to possess extensive common knowledge and powerful semantic comprehension abilities. We investigate two possible pipelines: LLMs-as-Enhancers and LLMs-as-Predictors.
  arXiv  Detail & Related papers  (2023-07-07T05:31:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.