Scaling Towards the Information Boundary of Instruction Set: InfinityInstruct-Subject Technical Report

• URL: http://arxiv.org/abs/2507.06968v1
• Date: Wed, 09 Jul 2025 15:59:02 GMT
• Title: Scaling Towards the Information Boundary of Instruction Set: InfinityInstruct-Subject Technical Report
• Authors: Li Du, Hanyu Zhao, Yiming Ju, Tengfei Pan,
• Abstract summary: Construction of high-quality instruction datasets is crucial for enhancing model performance and generalizability.<n>We propose a systematic instruction data synthesis framework, which integrates a hierarchical labeling system, an informative seed selection algorithm, and a model deficiency diagnosis.<n>Based on this framework, we construct InfinityInstruct-Subject, a high-quality dataset containing 1.5 million instructions.
• Score: 11.70656700216213
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Instruction tuning has become a foundation for unlocking the capabilities of large-scale pretrained models and improving their performance on complex tasks. Thus, the construction of high-quality instruction datasets is crucial for enhancing model performance and generalizability. Although current instruction datasets have reached tens of millions of samples, models finetuned on them may still struggle with complex instruction following and tasks in rare domains. This is primarily due to limited expansion in both ``coverage'' (coverage of task types and knowledge areas) and ``depth'' (instruction complexity) of the instruction set. To address this issue, we propose a systematic instruction data construction framework, which integrates a hierarchical labeling system, an informative seed selection algorithm, an evolutionary data synthesis process, and a model deficiency diagnosis with targeted data generation. These components form an iterative closed-loop to continuously enhance the coverage and depth of instruction data. Based on this framework, we construct InfinityInstruct-Subject, a high-quality dataset containing ~1.5 million instructions. Experiments on multiple foundation models and benchmark tasks demonstrate its effectiveness in improving instruction-following capabilities. Further analyses suggest that InfinityInstruct-Subject shows enlarged coverage and depth compared to comparable synthesized instruction datasets. Our work lays a theoretical and practical foundation for the efficient, continuous evolution of instruction datasets, moving from data quantity expansion to qualitative improvement.

Related papers

• From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding [33.009759731505746]
  We construct a dataset of 1 million instructions, called SynthQuestions.<n>We demonstrate that models trained on it achieve leading performance on several common benchmarks.
  arXiv  Detail & Related papers  (2025-06-04T14:00:47Z)
• Aligning Instruction Tuning with Pre-training [81.4748965653345]
  We propose Aligning Instruction Tuning with Pre-training (AITP) to align instruction tuning with pre-training distributions.<n>We show consistent performance improvements with AITP on three fully open large language models (LLMs) across eight benchmarks.
  arXiv  Detail & Related papers  (2025-01-16T08:27:40Z)
• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale [66.73529246309033]
  multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks.<n>Existing instruction-tuning datasets only provide phrase-level answers without any intermediate rationales.<n>We introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales.
  arXiv  Detail & Related papers  (2024-12-06T18:14:24Z)
• Optimizing Instruction Synthesis: Effective Exploration of Evolutionary Space with Tree Search [25.108044778194536]
  We introduce IDEA-MCTS (Instruction Data Enhancement using Monte Carlo Tree Search), a scalable framework for efficiently synthesizing instructions. With tree search and evaluation models, it can efficiently guide each instruction to evolve into a high-quality form, aiding in instruction fine-tuning. Experimental results show that IDEA-MCTS significantly enhances the seed instruction data, raising the average evaluation scores of quality, diversity, and complexity from 2.19 to 3.81.
  arXiv  Detail & Related papers  (2024-10-14T11:28:30Z)
• MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct [148.39859547619156]
  We propose MMEvol, a novel multimodal instruction data evolution framework.<n>MMEvol iteratively improves data quality through a refined combination of fine-grained perception, cognitive reasoning, and interaction evolution.<n>Our approach reaches state-of-the-art (SOTA) performance in nine tasks using significantly less data compared to state-of-the-art models.
  arXiv  Detail & Related papers  (2024-09-09T17:44:00Z)
• From Symbolic Tasks to Code Generation: Diversification Yields Better Task Performers [1.6958018695660049]
  We show that a more diverse instruction set, extending beyond code-related tasks, improves the performance of code generation. Our observations suggest that a more diverse semantic space for instruction-tuning sets greatly improves the model's ability to follow instructions and perform tasks.
  arXiv  Detail & Related papers  (2024-05-30T07:54:07Z)
• Mosaic-IT: Cost-Free Compositional Data Synthesis for Instruction Tuning [30.82220015525281]
  Mosaic Instruction Tuning (Mosaic-IT) is a human/model-free compositional data synthesis method.<n>Our evaluations demonstrate a superior performance and training efficiency of Mosaic-IT.
  arXiv  Detail & Related papers  (2024-05-22T04:08:20Z)
• Data Diversity Matters for Robust Instruction Tuning [129.83575908023312]
  Recent works have shown that by curating high quality and diverse instruction tuning datasets, we can significantly improve instruction-following capabilities. We propose a new algorithm, Quality-Diversity Instruction Tuning (QDIT) to control dataset diversity and quality. We validate the performance of QDIT on several large scale instruction tuning datasets, where we find it can substantially improve worst and average case performance.
  arXiv  Detail & Related papers  (2023-11-21T19:12:18Z)
• Dynamics of Instruction Fine-Tuning for Chinese Large Language Models [19.832906541004114]
  We systematically investigate the effects of data quantity, model size, and data construction methods on instruction tuning for Chinese Large Language Models.<n>Our experiments, conducted on models ranging from 7b to 33b parameters, yield three key findings.
  arXiv  Detail & Related papers  (2023-10-30T15:37:10Z)
• Homological Convolutional Neural Networks [4.615338063719135]
  We propose a novel deep learning architecture that exploits the data structural organization through topologically constrained network representations. We test our model on 18 benchmark datasets against 5 classic machine learning and 3 deep learning models.
  arXiv  Detail & Related papers  (2023-08-26T08:48:51Z)
• A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment [90.7443414448245]
  We propose Tree-Instruct to systematically enhance the instruction complexity in a controllable manner. By adding a specified number of nodes to instructions' semantic trees, this approach not only yields new instruction data but also allows us to control the difficulty level of modified instructions.
  arXiv  Detail & Related papers  (2023-08-10T16:58:51Z)
• Improving Long-Horizon Imitation Through Instruction Prediction [93.47416552953075]
  In this work, we explore the use of an often unused source of auxiliary supervision: language. Inspired by recent advances in transformer-based models, we train agents with an instruction prediction loss that encourages learning temporally extended representations that operate at a high level of abstraction. In further analysis we find that instruction modeling is most important for tasks that require complex reasoning, while understandably offering smaller gains in environments that require simple plans.
  arXiv  Detail & Related papers  (2023-06-21T20:47:23Z)
• Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation [92.2167864437497]
  We propose Dynosaur, a dynamic growth paradigm for the automatic curation of instruction-tuning data. Based on the metadata of existing datasets, we use LLMs to automatically construct instruction-tuning data by identifying relevant data fields and generating appropriate instructions. By leveraging the existing annotated datasets, Dynosaur offers several advantages: 1) it reduces the API cost for generating instructions; 2) it provides high-quality data for instruction tuning; and 3) it supports the continuous improvement of models by generating instruction-tuning data when a new annotated dataset becomes available.
  arXiv  Detail & Related papers  (2023-05-23T17:56:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.