RuleR: Improving LLM Controllability by Rule-based Data Recycling

• URL: http://arxiv.org/abs/2406.15938v3
• Date: Tue, 29 Oct 2024 14:28:24 GMT
• Title: RuleR: Improving LLM Controllability by Rule-based Data Recycling
• Authors: Ming Li, Han Chen, Chenguang Wang, Dang Nguyen, Dianqi Li, Tianyi Zhou,
• Abstract summary: Rule-based Data Recycling (RuleR) is a human/LLM-free data augmentation method incorporating multiple constraints into the original SFT data. RuleR integrates linguistic or formatting rules into the original instructions and modifies the responses to fulfill the rule-defined constraints. Experiments demonstrate RuleR's effectiveness in improving LLM controllability while maintaining general instruction-following performance.
• Score: 28.74786215922553
• License:
• Abstract: Despite the remarkable advancement of Large language models (LLMs), they still lack delicate controllability under sophisticated constraints, which is critical to enhancing their response quality and the user experience. While conditional supervised fine-tuning (SFT) can potentially improve LLM controllability, curating new SFT data to fulfill the constraints usually relies on human experts or proprietary LLMs, which is time-consuming and expensive. To bridge this gap, we propose Rule-based Data Recycling (RuleR), a human/LLM-free data augmentation method incorporating multiple constraints into the original SFT data. Instead of creating new responses from scratch, RuleR integrates linguistic or formatting rules into the original instructions and modifies the responses to fulfill the rule-defined constraints. Training on the "recycled" data consolidates LLMs capability to generate constrained outputs. Extensive experiments demonstrate RuleR's effectiveness in improving LLM controllability while maintaining general instruction-following performance. RuleR's code is released on https://github.com/tianyi-lab/RuleR.

Related papers

• Constraint Back-translation Improves Complex Instruction Following of Large Language Models [55.60192044049083]
  Large language models (LLMs) struggle to follow instructions with complex constraints in format, length, etc. Previous works conduct post-training on complex instruction-response pairs generated by feeding complex instructions to advanced LLMs. We propose a novel data generation technique, constraint back-translation.
  arXiv  Detail & Related papers  (2024-10-31T17:42:26Z)
• LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints [86.59857711385833]
  We introduce RealInstruct, the first benchmark designed to evaluate LLMs' ability to follow real-world multi-constrained instructions. To address the performance gap between open-source and proprietary models, we propose the Decompose, Critique and Refine (DeCRIM) self-correction pipeline. Our results show that DeCRIM improves Mistral's performance by 7.3% on RealInstruct and 8.0% on IFEval even with weak feedback.
  arXiv  Detail & Related papers  (2024-10-09T01:25:10Z)
• Open-domain Implicit Format Control for Large Language Model Generation [52.83173553689678]
  We introduce a novel framework for controlled generation in large language models (LLMs) This study investigates LLMs' capabilities to follow open-domain, one-shot constraints and replicate the format of the example answers. We also develop a dataset collection methodology for supervised fine-tuning that enhances the open-domain format control of LLMs without degrading output quality.
  arXiv  Detail & Related papers  (2024-08-08T11:51:45Z)
• DELRec: Distilling Sequential Pattern to Enhance LLM-based Recommendation [3.5113201254928117]
  Sequential recommendation (SR) tasks enhance recommendation accuracy by capturing the connection between users' past interactions and their changing preferences. Conventional models often focus solely on capturing sequential patterns within the training data, neglecting the broader context and semantic information embedded in item titles from external sources. DelRec aims to extract knowledge from SR models and enable LLMs to easily comprehend and utilize this supplementary information for more effective sequential recommendations.
  arXiv  Detail & Related papers  (2024-06-17T02:47:09Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs) We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes [57.62036621319563]
  We introduce CLLM, which leverages the prior knowledge of Large Language Models (LLMs) for data augmentation in the low-data regime. We demonstrate the superior performance of CLLM in the low-data regime compared to conventional generators.
  arXiv  Detail & Related papers  (2023-12-19T12:34:46Z)
• Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning [79.32236399694077]
  Low-quality data in the training set are usually detrimental to instruction tuning. We propose a novel method, termed "reflection-tuning" This approach utilizes an oracle LLM to recycle the original training data by introspecting and enhancing the quality of instructions and responses in the data.
  arXiv  Detail & Related papers  (2023-10-18T05:13:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.