Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMs

• URL: http://arxiv.org/abs/2603.29232v1
• Date: Tue, 31 Mar 2026 04:03:07 GMT
• Title: Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMs
• Abstract summary: Large language models (LLMs) are widely applied to data analytics over documents, yet direct reasoning over long, noisy documents remains brittle and error-prone.<n>We propose a two-pillar framework, LiteCoST, to achieve both high accuracy and low latency with small language models (SLMs)
• Score: 15.488953497866525
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are widely applied to data analytics over documents, yet direct reasoning over long, noisy documents remains brittle and error-prone. Hence, we study document question answering (QA) that consolidates dispersed evidence into a structured output (e.g., a table, graph, or chunks) to support reliable, verifiable QA. We propose a two-pillar framework, LiteCoST, to achieve both high accuracy and low latency with small language models (SLMs). Pillar 1: Chain-of-Structured-Thought (CoST). We introduce a CoST template, a schema-aware instruction that guides a strong LLM to produce both a step-wise CoST trace and the corresponding structured output. The process induces a minimal structure, normalizes entities/units, aligns records, serializes the output, and verifies/refines it, yielding auditable supervision. Pillar 2: SLM fine-tuning. The compact models are trained on LLM-generated CoST data in two stages: Supervised Fine-Tuning for structural alignment, followed by Group Relative Policy Optimization (GRPO) incorporating triple rewards for answer/format quality and process consistency. By distilling structure-first behavior into SLMs, this approach achieves LLM-comparable quality on multi-domain long-document QA using 3B/7B SLMs, while delivering 2-4x lower latency than GPT-4o and DeepSeek-R1 (671B). The code is available at https://github.com/HKUSTDial/LiteCoST.

Related papers

• Align Documents to Questions: Question-Oriented Document Rewriting for Retrieval-Augmented Generation [51.55755193937205]
  We propose QREAM, a style-controlled rewriter that aligns retrieved documents with a question-oriented style while preserving facts.<n>Our framework consists of two stages: (1) QREAM-ICL, which uses stylistic seeds to guide iterative rewriting exploration; and (2) QREAM-FT, a lightweight student model distilled from denoised ICL outputs.
  arXiv  Detail & Related papers  (2026-04-19T08:39:21Z)
• DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding [63.257540233507626]
  We propose a paradigm that requires the model to execute a structured Analysis, Localization and Reasoning workflow.<n>We show it robustly generalizes from short-page training to ultra-long documents and is naturally synergistic with visual Retrieval-Augmented Generation systems.
  arXiv  Detail & Related papers  (2026-04-14T14:39:26Z)
• HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy [54.91468501159335]
  Cross-document relation extraction (RE) aims to identify relations between the head tail entities located in different documents.<n>We propose an inference strategy that improves reliability through multi-view verification at each level.
  arXiv  Detail & Related papers  (2026-04-09T07:55:27Z)
• Self-Correction Distillation for Structured Data Question Answering [50.98882432829651]
  Small-scale language models (LLMs) are prone to errors in generating structured queries.<n>We propose a self-correction distillation (SCD) method to improve the structured data QA ability of small-scale LLMs.
  arXiv  Detail & Related papers  (2025-11-11T09:01:51Z)
• DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correction [29.22321321753093]
  Large language models (LLMs) often prioritize reasoning over adherence to detailed instructions.<n>Fine-tuning LLMs on supervised datasets to address this is impractical due to high computational costs and limited parameter access.<n>We propose DICE, a lightweight framework that guides small language models (SLMs) to refine LLMs' outputs through chain-of-thought (CoT) correction.
  arXiv  Detail & Related papers  (2025-10-10T09:45:35Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• LLMSR@XLLM25: Less is More: Enhancing Structured Multi-Agent Reasoning via Quality-Guided Distillation [6.920352059545929]
  We present Less is More, the third-place winning approach in the LLMSR@XLLM25 structural reasoning task.<n>Our approach leverages a multi-agent framework with reverse-prompt induction, retrieval-augmented reasoning synthesis via GPT-4o, and dual-stage reward-guided filtering.<n>All modules are fine-tuned from Meta-Llama-3-8B-Instruct under a unified LoRA+ setup.
  arXiv  Detail & Related papers  (2025-04-23T04:19:52Z)
• Improving Consistency in Large Language Models through Chain of Guidance [9.040736633675136]
  Chain of Guidance (CoG) is a multistep prompting technique that generates highly consistent outputs from Large Language Models (LLMs)<n>We use synthetic data sets comprised of consistent input-output pairs to fine-tune LLMs to produce consistent and correct outputs.<n>Our fine-tuned models are more than twice as consistent compared to base models and show strong generalization capabilities by producing consistent outputs over datasets not used in the fine-tuning process.
  arXiv  Detail & Related papers  (2025-02-21T20:41:37Z)
• PECAN: LLM-Guided Dynamic Progress Control with Attention-Guided Hierarchical Weighted Graph for Long-Document QA [16.945257645760428]
  Long-document QA presents challenges with large-scale text and long-distance dependencies.<n>Recent advances in Large Language Models (LLMs) enable entire documents to be processed in a single pass.<n>We propose a novel retrieval method that constructs a hierarchical graph where edges are derived by LLM attention weights.
  arXiv  Detail & Related papers  (2024-10-07T07:02:09Z)
• Open-domain Implicit Format Control for Large Language Model Generation [52.83173553689678]
  We introduce a novel framework for controlled generation in large language models (LLMs) This study investigates LLMs' capabilities to follow open-domain, one-shot constraints and replicate the format of the example answers. We also develop a dataset collection methodology for supervised fine-tuning that enhances the open-domain format control of LLMs without degrading output quality.
  arXiv  Detail & Related papers  (2024-08-08T11:51:45Z)
• TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data [73.29220562541204]
  We consider harnessing the amazing power of language models (LLMs) to solve our task. We develop a TAT-LLM language model by fine-tuning LLaMA 2 with the training data generated automatically from existing expert-annotated datasets.
  arXiv  Detail & Related papers  (2024-01-24T04:28:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.