Step-Back Profiling: Distilling User History for Personalized Scientific Writing
- URL: http://arxiv.org/abs/2406.14275v2
- Date: Thu, 11 Jul 2024 07:29:12 GMT
- Title: Step-Back Profiling: Distilling User History for Personalized Scientific Writing
- Authors: Xiangru Tang, Xingyao Zhang, Yanjun Shao, Jie Wu, Yilun Zhao, Arman Cohan, Ming Gong, Dongmei Zhang, Mark Gerstein,
- Abstract summary: Large language models (LLM) excel at a variety of natural language processing tasks, yet they struggle to generate personalized content for individuals.
We introduce STEP-BACK PROFILING to personalize LLMs by distilling user history into concise profiles.
Our approach outperforms the baselines by up to 3.6 points on the general personalization benchmark.
- Score: 50.481041470669766
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Large language models (LLM) excel at a variety of natural language processing tasks, yet they struggle to generate personalized content for individuals, particularly in real-world scenarios like scientific writing. Addressing this challenge, we introduce STEP-BACK PROFILING to personalize LLMs by distilling user history into concise profiles, including essential traits and preferences of users. To conduct the experiments, we construct a Personalized Scientific Writing (PSW) dataset to study multi-user personalization. PSW requires the models to write scientific papers given specialized author groups with diverse academic backgrounds. As for the results, we demonstrate the effectiveness of capturing user characteristics via STEP-BACK PROFILING for collaborative writing. Moreover, our approach outperforms the baselines by up to 3.6 points on the general personalization benchmark (LaMP), including 7 personalization LLM tasks. Our ablation studies validate the contributions of different components in our method and provide insights into our task definition. Our dataset and code are available at \url{https://github.com/gersteinlab/step-back-profiling}.
Related papers
- SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts.
We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM.
We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
arXiv Detail & Related papers (2024-07-16T04:41:58Z) - Evaluating Character Understanding of Large Language Models via Character Profiling from Fictional Works [33.817319226631426]
Large language models (LLMs) have demonstrated impressive performance and spurred numerous AI applications.
The prerequisite for these RPAs lies in the capability of LLMs to understand characters from fictional works.
Previous efforts have evaluated this capability via basic classification tasks or characteristic imitation.
arXiv Detail & Related papers (2024-04-19T09:10:29Z) - Editing Personality for Large Language Models [73.59001811199823]
This paper introduces an innovative task focused on editing the personality traits of Large Language Models (LLMs)
We construct a new benchmark dataset PersonalityEdit to address this task.
arXiv Detail & Related papers (2023-10-03T16:02:36Z) - PerPLM: Personalized Fine-tuning of Pretrained Language Models via
Writer-specific Intermediate Learning and Prompts [16.59511985633798]
Pretrained language models (PLMs) are powerful tools for capturing context.
PLMs are typically pretrained and fine-tuned for universal use across different writers.
This study aims to improve the accuracy of text understanding tasks by personalizing the fine-tuning of PLMs for specific writers.
arXiv Detail & Related papers (2023-09-14T14:03:48Z) - Teach LLMs to Personalize -- An Approach inspired by Writing Education [37.198598706659524]
We propose a general approach for personalized text generation using large language models (LLMs)
Inspired by the practice of writing education, we develop a multistage and multitask framework to teach LLMs for personalized generation.
arXiv Detail & Related papers (2023-08-15T18:06:23Z) - Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A
Preliminary Study on Writing Assistance [60.40541387785977]
Small foundational models can display remarkable proficiency in tackling diverse tasks when fine-tuned using instruction-driven data.
In this work, we investigate a practical problem setting where the primary focus is on one or a few particular tasks rather than general-purpose instruction following.
Experimental results show that fine-tuning LLaMA on writing instruction data significantly improves its ability on writing tasks.
arXiv Detail & Related papers (2023-05-22T16:56:44Z) - LaMP: When Large Language Models Meet Personalization [35.813652110400064]
This paper highlights the importance of personalization in large language models and introduces the LaMP benchmark.
LaMP is a novel benchmark for training and evaluating language models for producing personalized outputs.
arXiv Detail & Related papers (2023-04-22T13:42:04Z) - Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable
Reward Function [19.652303125864204]
We propose a novel framework, P-ToD, to personalize task-oriented dialog systems.
P-ToD uses a pre-trained GPT-2 as a backbone model and works in three phases.
Our novel reward function can quantify the quality of the generated responses even for unseen profiles.
arXiv Detail & Related papers (2023-03-24T04:33:40Z) - Unsupervised Neural Stylistic Text Generation using Transfer learning
and Adapters [66.17039929803933]
We propose a novel transfer learning framework which updates only $0.3%$ of model parameters to learn style specific attributes for response generation.
We learn style specific attributes from the PERSONALITY-CAPTIONS dataset.
arXiv Detail & Related papers (2022-10-07T00:09:22Z) - PART: Pre-trained Authorship Representation Transformer [64.78260098263489]
Authors writing documents imprint identifying information within their texts: vocabulary, registry, punctuation, misspellings, or even emoji usage.
Previous works use hand-crafted features or classification tasks to train their authorship models, leading to poor performance on out-of-domain authors.
We propose a contrastively trained model fit to learn textbfauthorship embeddings instead of semantics.
arXiv Detail & Related papers (2022-09-30T11:08:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.