Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and Benchmark Datasets from Industrial Technical Reports
- URL: http://arxiv.org/abs/2608.22817v1
- Date: Mon, 24 Aug 2026 05:28:25 GMT
- Title: Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and Benchmark Datasets from Industrial Technical Reports
- Abstract summary: Industrial technical reports contain high-value knowledge for maintenance, troubleshooting, and product engineering.<n>No public instruction-tuning or benchmark datasets are built from such documents.<n>We contribute two open QA datasets built from real industrial technical reports and the end-to-end pipeline that produces them.
- Score: 1.806188080472605
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Industrial technical reports contain high-value knowledge for maintenance, troubleshooting, and product engineering, but their heterogeneous structure (dense prose, specifications, tables) makes them difficult to index and reason over with standard retrieval and QA pipelines, and no public instruction-tuning or benchmark datasets are built from such documents. We address this gap with Industrial-Instruction, contributing (i) two open QA datasets built from real industrial technical reports and (ii) the end-to-end pipeline that produces them. Using 906 public Panasonic documents (7,525 pages), we apply layout-aware extraction, build a semantic retrieval index, and synthesize multiple-choice QA grounded in retrieved evidence under five query-document relationships (irrelevant retrieval, single-/multi-document support, single-/multi-document answer). After filtering an initial 23.9k generated samples, each dataset provides approximately 13.6k QA pairs with source documents and a held-out benchmark split. Fine-tuning small open LLMs (under 10B parameters) improves Set-Match Accuracy from 28.5% to 42.0% and F1 from 46.6% to 63.5% on the Panasonic benchmark. We release two parallel versions built by the same pipeline: one generated with the open-weight Qwen3-30B-A3B-Instruct model and one with the closed, API-based Claude-Opus-4.6 model, enabling a direct comparison of open- versus frontier-model data generation. The Claude-Opus-4.6 dataset yields a cleaner raw corpus and larger fine-tuning gains, at roughly two orders of magnitude higher cost. MMLU evaluation shows models trained on the Claude-Opus-4.6 data retain essentially all general knowledge, versus a small but measurable forgetting effect for the Qwen-generated data. Together, these datasets and pipeline offer a practical, reproducible path toward scalable industrial benchmarks and training data from real-world documentation.
Related papers
- A Hybrid Vision-Language Architecture for Automated Defect Reasoning and Report Generation in Industrial Inspection [0.42970700836450487]
This paper describes a decoupled, edge-deployable pipeline for wind turbine blade inspection.<n>The Eyes a Y26-x-obb oriented bounding-box localizes defects at dataset-native resolution.<n>The Bridge a deterministic, parameter-free encoding module maps each detected bounding box to grid-referenced tokens.
arXiv Detail & Related papers (2026-05-26T04:27:38Z) - How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings [56.70440596502351]
The past year has seen over 20 open document parsing models, yet benchmarks almost exclusively on OmniDocBench.<n>We present PureDocBench, a benchmark thatrenders document images from HTML/CSS, covering 10 domains, 66 pages, and 1,475 pages.
arXiv Detail & Related papers (2026-05-08T09:30:31Z) - MinerU2.5-Pro: Pushing the Limits of Data-Centric Document Parsing at Scale [92.09717763663873]
We present MinerU2.5-Pro, which advances the state of the art purely through data engineering and training strategy design.<n>At its core is a Data Engine co-designed around coverage, informativeness, and annotation accuracy.<n>We show that MinerU2.5-Pro achieves 95.69 on OmniDocBench v1.6, improving over the same-architecture baseline by 2.71 points and surpassing all existing methods.
arXiv Detail & Related papers (2026-04-06T15:44:18Z) - VAREX: A Benchmark for Multi-Modal Structured Extraction from Documents [1.06378109904813]
VAREX is a benchmark for evaluating structured data extraction from government forms.<n>The benchmark comprises 1,777 documents with 1,771 unique documents with validated ground truth through three-phase quality assurance.<n>Results reveal that below 4B parameters, compliance output -- not extraction capability -- is a dominant bottleneck.
arXiv Detail & Related papers (2026-03-16T11:15:56Z) - Engineering Reasoning and Instruction (ERI) Benchmark: A Large Taxonomy-driven Dataset for Foundation Models and Agents [1.629288881045104]
The Engineering Reasoning and Instruction (ERI) benchmark is a taxonomy-driven instruction dataset designed to train and evaluate engineeringcapable large language models (LLMs) and agents.<n>This dataset spans nine engineering fields (civil, mechanical, electrical, chemical, environmental, aerospace, materials, fire, and industrial engineering) and 55, and is crossed with seven intent types (i.e., definition, explanation, calculation, comparison, design/synthesis, troubleshooting, and code-related) and three difficulty tiers (undergraduate, graduate, and professional)<n>ERI is released with taxonomy specifications, validation scripts, and an evaluation harness to
arXiv Detail & Related papers (2026-02-16T12:38:08Z) - OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value [74.80873109856563]
OpenDataArena (ODA) is a holistic and open platform designed to benchmark the intrinsic value of post-training data.<n>ODA establishes a comprehensive ecosystem comprising four key pillars: (i) a unified training-evaluation pipeline that ensures fair, open comparisons across diverse models; (ii) a multi-dimensional scoring framework that profiles data quality along tens of distinct axes; and (iii) an interactive data lineage explorer to visualize dataset genealogy and dissect component sources.
arXiv Detail & Related papers (2025-12-16T03:33:24Z) - Generating High-Quality Datasets for Code Editing via Open-Source Language Models [49.01026966779838]
We introduce OpenCodeEdit, an open-source pipeline that synthesizes realistic code-edit triplets.<n>The pipeline produces both concise "lazy" instructions and more detailed "descriptive" ones, and applies filtering based on diffs and topics to guarantee data quality and variety.<n>Fine-tuning three advanced base models on OCEDataFT leads to significant performance boosts on the CanItEdit benchmark, with relative pass@1 improvements ranging from 4.50% to 20.79%.
arXiv Detail & Related papers (2025-09-19T03:57:39Z) - Leveraging Generative Models for Real-Time Query-Driven Text Summarization in Large-Scale Web Search [54.987957691350665]
Query-Driven Text Summarization (QDTS) aims to generate concise and informative summaries from textual documents based on a given query.<n>Traditional extractive summarization models, based primarily on ranking candidate summary segments, have been the dominant approach in industrial applications.<n>We propose a novel framework to pioneer the application of generative models to address real-time QDTS in industrial web search.
arXiv Detail & Related papers (2025-08-28T08:51:51Z) - MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm [60.14048367611333]
MonkeyOCR is a vision-language model for document parsing.<n>It advances the state of the art by leveraging a Structure-Recognition-Relation (SRR) triplet paradigm.
arXiv Detail & Related papers (2025-06-05T16:34:57Z) - LLMs to Support a Domain Specific Knowledge Assistant [0.0]
This work presents a custom approach to developing a domain specific knowledge assistant for sustainability reporting using the International Financial Reporting Standards (IFRS)<n>In this domain, there is no publicly available question-answer dataset, which has impeded the development of a high-quality pipeline to support companies with reporting.
arXiv Detail & Related papers (2025-02-06T14:12:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.