Unsupervised Human Preference Learning

• URL: http://arxiv.org/abs/2410.03731v3
• Date: Fri, 11 Oct 2024 18:53:56 GMT
• Title: Unsupervised Human Preference Learning
• Authors: Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Dilek Hakkani-Tür,
• Abstract summary: Large language models demonstrate impressive reasoning abilities but struggle to provide personalized content. Existing methods, such as in-context learning and parameter-efficient fine-tuning, fall short in capturing the complexity of human preferences. We propose a novel approach utilizing small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model.
• Score: 7.959043497459107
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models demonstrate impressive reasoning abilities but struggle to provide personalized content due to their lack of individual user preference information. Existing methods, such as in-context learning and parameter-efficient fine-tuning, fall short in capturing the complexity of human preferences, especially given the small, personal datasets individuals possess. In this paper, we propose a novel approach utilizing small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model, enabling efficient personalization. Our method involves a small, local "steering wheel" model that directs the outputs of a much larger foundation model, producing content tailored to an individual's preferences while leveraging the extensive knowledge and capabilities of the large model. Importantly, this personalization is achieved without the need to fine-tune the large model. Experimental results on email and article datasets, demonstrate that our technique significantly outperforms baseline personalization methods. By allowing foundation models to adapt to individual preferences in a data and compute-efficient manner, our approach paves the way for highly personalized language model applications.

Related papers

• Investigating on RLHF methodology [0.0]
  We discuss the features of training a Preference Model, which simulates human preferences, and the methods and details we found essential for achieving the best results. We also discuss using Reinforcement Learning to fine-tune Large Language Models and describe the challenges we faced and the ways to overcome them.
  arXiv  Detail & Related papers  (2024-10-02T17:46:22Z)
• Aligning Large Language Models with Self-generated Preference Data [72.99676237703099]
  We propose a new framework that boosts the alignment of large language models (LLMs) with human preferences. Our key idea is leveraging the human prior knowledge within the small (seed) data. We introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data.
  arXiv  Detail & Related papers  (2024-06-06T18:01:02Z)
• Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond [87.1712108247199]
  Our goal is to establish a Unified paradigm for Multi-modal Personalization systems (UniMP) We develop a generic and personalization generative framework, that can handle a wide range of personalized needs. Our methodology enhances the capabilities of foundational language models for personalized tasks.
  arXiv  Detail & Related papers  (2024-03-15T20:21:31Z)
• Personalized Language Modeling from Personalized Human Feedback [49.344833339240566]
  Reinforcement Learning from Human Feedback (RLHF) is commonly used to fine-tune large language models to better align with human preferences. In this work, we aim to address this problem by developing methods for building personalized language models.
  arXiv  Detail & Related papers  (2024-02-06T04:18:58Z)
• When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunities [60.5609416496429]
  The capability of large language models has been dramatically improved. Such a major leap-forward in general AI capacity will change the pattern of how personalization is conducted. By leveraging large language models as general-purpose interface, personalization systems may compile user requests into plans.
  arXiv  Detail & Related papers  (2023-07-31T02:48:56Z)
• Selective Pre-training for Private Fine-tuning [33.55628974557588]
  We show that a careful pre-training on a public dataset is crucial to train small language models with differential privacy. Results demonstrate that smaller models, through careful pre-training and private fine-tuning, can match the performance of much larger models that do not have access to private data.
  arXiv  Detail & Related papers  (2023-05-23T09:36:58Z)
• Can Public Large Language Models Help Private Cross-device Federated Learning? [58.05449579773249]
  We study (differentially) private federated learning (FL) of language models. Public data has been used to improve privacy-utility trade-offs for both large and small language models. We propose a novel distribution matching algorithm with theoretical grounding to sample public data close to private data distribution.
  arXiv  Detail & Related papers  (2023-05-20T07:55:58Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Self-Supervised Learning for Personalized Speech Enhancement [25.05285328404576]
  Speech enhancement systems can show improved performance by adapting the model towards a single test-time speaker. Test-time user might only provide a small amount of noise-free speech data, likely insufficient for traditional fully-supervised learning. We propose self-supervised methods that are designed specifically to learn personalized and discriminative features from abundant in-the-wild noisy, but still personal speech recordings.
  arXiv  Detail & Related papers  (2021-04-05T17:12:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.