IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking

• URL: http://arxiv.org/abs/2606.09709v1
• Date: Mon, 08 Jun 2026 16:31:00 GMT
• Title: IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking
• Abstract summary: We introduce the Interleaved Structural Chain-of-Thought (IS-CoT) framework.<n>IS-CoT embeds a dynamic Plan-Write-Reflect cycle into the generation process.<n>We construct a high-quality dataset of interleaved reasoning traces via a multi-teacher pipeline and train IS-Writer-8B.
• Score: 50.381430711141995
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Generating coherent and controllable long-form content remains a persistent challenge for Large Language Models (LLMs). While reasoning-enhanced models have demonstrated success in logic-intensive domains, our evaluation reveals that they suffer from a severe length collapse in open-ended writing, where performance degrades sharply as target lengths exceed 2,000 words. We attribute this failure to the limitation of static hierarchical planning, which struggles to provide dynamic guidance over extended contexts. To bridge this gap, we introduce the Interleaved Structural Chain-of-Thought (IS-CoT) framework. Unlike external agentic workflows, IS-CoT embeds a dynamic Plan-Write-Reflect cycle into the generation process, enabling continuous strategy adaptation and global alignment without additional assistance. Based on this framework, we construct a high-quality dataset of interleaved reasoning traces via a multi-teacher pipeline and train IS-Writer-8B. Experiments demonstrate that IS-Writer-8B achieves state-of-the-art performance on challenging long-form benchmarks (e.g., +3.08 vs. DeepSeek-V3.2 on LongBench-Write), exhibiting robust length compliance and coherence competitive with significantly larger proprietary models.

Related papers

• LongCrafter: Towards Diverse Long-Context Understanding via Evidence-Graph-Guided Instruction Synthesis [25.64387650835006]
  LongCrafter is a structured synthesis framework that couples a hierarchical task taxonomy with an evidence-grounded pipeline.<n>LongCrafter constructs task-aligned long contexts, decomposes them into explicit evidence graphs that model cross-paragraph dependencies, and generates instruction--response pairs strictly grounded in the located evidence spans.<n>Models fine-tuned on LongCrafter data outperform all SFT baselines and even the official post-trained models.
  arXiv  Detail & Related papers  (2026-07-07T11:35:48Z)
• Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference [51.12849550784653]
  Diffusion language models (DLMs) have emerged as a promising alternative to autoregressive models.<n>Most existing DLMs rely on a fixed generation length specified prior to decoding, which restricts their flexibility in real-world applications.<n>We propose a training-free, Bayesian structured decoding framework that formulates flexible-length generation as a dynamic structural inference problem.
  arXiv  Detail & Related papers  (2026-05-10T23:51:14Z)
• Constraint Decay: The Fragility of LLM Agents in Backend Code Generation [9.659020624935687]
  Large Language Model (LLM) agents demonstrate strong performance in autonomous code generation under loose specifications.<n>Production-grade software requires strict adherence to structural constraints, such as architectural patterns, databases, and object-relational mappings.<n>We present a systematic study evaluating how well agents handle structural constraints in backend generation.
  arXiv  Detail & Related papers  (2026-05-07T15:44:40Z)
• LongR: Unleashing Long-Context Reasoning via Reinforcement Learning with Dense Utility Rewards [57.993003392037174]
  LongR is a framework that enhances long-context performance by integrating a dynamic "Think-and-Read" mechanism.<n>LongR consistently enhances performance across diverse RL algorithms.
  arXiv  Detail & Related papers  (2026-02-05T15:26:47Z)
• Short Chains, Deep Thoughts: Balancing Reasoning Efficiency and Intra-Segment Capability via Split-Merge Optimization [68.89915707647138]
  Large Reasoning Models (LRMs) have demonstrated impressive capabilities in solving complex tasks through the generation of long reasoning chains.<n>We propose textbfCoSMo (textbfSplit-textbfMerge textbfOptimization), a framework designed to eliminate structural redundancy rather than indiscriminately restricting token volume.
  arXiv  Detail & Related papers  (2026-02-03T05:54:28Z)
• CCF: A Context Compression Framework for Efficient Long-Sequence Language Modeling [52.05149789178508]
  CCF is a novel context compression framework designed to enable efficient long-context modeling.<n>CCF integrates segment-wise semantic aggregation with key-value memory encoding, forming compact representations.<n> Empirical results on multiple long-context language modeling benchmarks demonstrate that CCF achieves competitive perplexity under high compression ratios.
  arXiv  Detail & Related papers  (2025-09-11T07:13:49Z)
• Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models [74.15250326312179]
  Diffusion Large Language Models offer efficient parallel generation and capable global modeling.<n>The dominant application ofDLLMs is hindered by the need for a statically predefined generation length.<n>We introduce DAEDAL, a novel training-free denoising strategy that enables Dynamic Adaptive Length Expansion.
  arXiv  Detail & Related papers  (2025-08-01T17:56:07Z)
• Long-Short Alignment for Effective Long-Context Modeling in LLMs [32.13785291956956]
  Large language models (LLMs) have exhibited impressive performance and surprising emergent properties.<n>Length generalization -- the ability to generalize to sequences longer than those seen during training -- is a classical and fundamental problem.<n>We highlight the critical role of textbflong-short alignment -- the consistency of output distributions across sequences of varying lengths.
  arXiv  Detail & Related papers  (2025-06-13T13:25:39Z)
• SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models [34.723917246316205]
  SuperWriter-Agent is a framework designed to enhance the quality and consistency of long-form text generation.<n>Based on this framework, we construct a supervised fine-tuning dataset to train a 7B SuperWriter-LM.<n> Empirical results across diverse benchmarks demonstrate that SuperWriter-LM achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-06-04T17:27:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.