RNR: Teaching Large Language Models to Follow Roles and Rules

• URL: http://arxiv.org/abs/2409.13733v1
• Date: Tue, 10 Sep 2024 06:07:32 GMT
• Title: RNR: Teaching Large Language Models to Follow Roles and Rules
• Authors: Kuan Wang, Alexander Bukharin, Haoming Jiang, Qingyu Yin, Zhengyang Wang, Tuo Zhao, Jingbo Shang, Chao Zhang, Bing Yin, Xian Li, Jianshu Chen, Shiyang Li,
• Abstract summary: We propose model, an automated data generation pipeline that generates diverse roles and rules from existing IFT instructions. This data can then be used to train models that follow complex system prompts. Our framework significantly improves role and rule following capability in large language models.
• Score: 153.6596303205894
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Instruction fine-tuning (IFT) elicits instruction following capabilities and steers the behavior of large language models (LLMs) via supervised learning. However, existing models trained on open-source IFT datasets only have the ability to follow instructions from users, and often fail to follow complex role and rules specified by developers, a.k.a. system prompts. The ability to follow these roles and rules is essential for deployment, as it ensures that the model safely interacts with users within developer defined guidelines. To improve such role and rule following ability, we propose \model, an automated data generation pipeline that generates diverse roles and rules from existing IFT instructions, along with corresponding responses. This data can then be used to train models that follow complex system prompts. The models are evaluated on our newly created benchmarks for role and rule following ability, as well as standard instruction-following benchmarks and general NLP tasks. Our framework significantly improves role and rule following capability in LLMs, as evidenced by over 25% increase in pass-rate on rule adherence, i.e. following all requirements, in our experiments with the Alpaca and Ultrachat datasets. Moreover, our models achieves this increase without any regression on popular instruction following benchmarks.

Related papers

• Balancing Continuous Pre-Training and Instruction Fine-Tuning: Optimizing Instruction-Following in LLMs [4.096028601599825]
  Large Language Models (LLMs) for public use require continuous pre-training to remain up-to-date with the latest data. This study aims to find the most compute-efficient strategy to gain up-to-date knowledge and instruction-following capabilities without requiring any instruction data and fine-tuning.
  arXiv  Detail & Related papers  (2024-10-14T17:20:30Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• From Instance Training to Instruction Learning: Task Adapters Generation from Instructions [29.452006810725184]
  This paper focuses on simulating human learning to address the shortcomings of instance training. We introduce Task Adapters Generation from Instructions (TAGI), which automatically constructs the task-specific model. We evaluate TAGI on the Super-Natural Instructions and P3 datasets.
  arXiv  Detail & Related papers  (2024-06-18T08:14:28Z)
• FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions [71.5977045423177]
  We study the use of instructions in Information Retrieval systems. We introduce our dataset FollowIR, which contains a rigorous instruction evaluation benchmark. We show that it is possible for IR models to learn to follow complex instructions.
  arXiv  Detail & Related papers  (2024-03-22T14:42:29Z)
• TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning [7.9961739811640244]
  Development of Large Language Models often confronts challenges stemming from heavy reliance on human annotators. In this work, we pivot to Reinforcement Learning -- but with a twist. We use RL to directly generate the foundational instruction dataset that alone suffices for fine-tuning.
  arXiv  Detail & Related papers  (2024-03-13T16:57:57Z)
• Enhancing Role-playing Systems through Aggressive Queries: Evaluation and Improvement [17.5855800570993]
  Large Language Models (LLMs) have propelled dialogue generation into new realms, particularly in the field of role-playing systems (RPSs) Existing LLM-based RPSs still struggle to align with roles when handling intricate and trapped queries in boundary scenarios. We design the Modular ORchestrated Trap-setting Interaction SystEm (MORTISE) to benchmark and improve the role-playing LLMs' performance.
  arXiv  Detail & Related papers  (2024-02-16T12:12:05Z)
• Tuna: Instruction Tuning using Feedback from Large Language Models [74.04950416204551]
  We propose finetuning an instruction-tuned large language model using our novel textitprobabilistic ranking and textitcontextual ranking approaches. Probabilistic ranking enables the instruction-tuned model to inherit the relative rankings of high-quality and low-quality responses from the teacher LLM. On the other hand, learning with contextual ranking allows the model to refine its own response distribution using the contextual understanding ability of stronger LLMs.
  arXiv  Detail & Related papers  (2023-10-20T09:55:06Z)
• SALMON: Self-Alignment with Instructable Reward Models [80.83323636730341]
  This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision. We develop an AI assistant named Dromedary-2 with only 6 exemplars for in-context learning and 31 human-defined principles.
  arXiv  Detail & Related papers  (2023-10-09T17:56:53Z)
• RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models [107.00832724504752]
  We introduce RoleLLM, a framework to benchmark, elicit, and enhance role-playing abilities in Large Language Models (LLMs) By Context-Instruct and RoleGPT, we create RoleBench, the first systematic and fine-grained character-level benchmark dataset for role-playing with 168,093 samples.
  arXiv  Detail & Related papers  (2023-10-01T17:52:59Z)
• ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation [43.270424225285105]
  We focus on adapting and empowering a pure large language model for zero-shot and few-shot recommendation tasks. We propose Retrieval-enhanced Large Language models (ReLLa) for recommendation tasks in both zero-shot and few-shot settings.
  arXiv  Detail & Related papers  (2023-08-22T02:25:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.