Pref-CTRL: Preference Driven LLM Alignment using Representation Editing

• URL: http://arxiv.org/abs/2604.23543v1
• Date: Sun, 26 Apr 2026 05:41:40 GMT
• Title: Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
• Abstract summary: Test-time alignment methods offer a promising alternative to fine-tuning large language models (LLMs) at inference time.<n>We propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data.<n>Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets.
• Score: 5.327210719701061
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Test-time alignment methods offer a promising alternative to fine-tuning by steering the outputs of large language models (LLMs) at inference time with lightweight interventions on their internal representations. Recently, a prominent and effective approach, RE-Control (Kong et al., 2024), has proposed leveraging an external value function trained over the LLM's hidden states to guide generation via gradient-based editing. While effective, this method overlooks a key characteristic of alignment tasks, i.e. that they are typically formulated as learning from human preferences between candidate responses. To address this, in this paper we propose a novel preference-based training framework, Pref-CTRL, that uses a multi-objective value function to better reflect the structure of preference data. Our approach has outperformed RE-Control on two benchmark datasets and showed greater generalization on out-of-domain datasets. Our source code is available at https://github.com/UTS-nlPUG/pref-ctrl.

Related papers

• Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback [40.01227095901647]
  Large language models (LLMs) demonstrate impressive performance but lack the flexibility to adapt to human preferences quickly without retraining.<n>We introduce Test-time Preference Optimization (TPO), a framework that aligns LLM outputs with human preferences during inference.<n>Our findings establish TPO as a practical, lightweight alternative for test-time preference optimization, achieving alignment on the fly.
  arXiv  Detail & Related papers  (2025-01-22T14:15:46Z)
• Online Preference Alignment for Language Models via Count-based Exploration [46.46627519343809]
  Reinforcement Learning from Human Feedback (RLHF) has shown great potential in fine-tuning Large Language Models (LLMs) to align with human preferences.<n>Existing methods perform preference alignment from a fixed dataset, which can be limited in data coverage.<n>Online RLHF is more desirable to empower the LLM to explore outside the support of the initial dataset by iteratively collecting the prompt-response pairs.
  arXiv  Detail & Related papers  (2025-01-22T09:12:09Z)
• Preference-Guided Reflective Sampling for Aligning Language Models [27.69410513313001]
  Iterative data generation and model re-training can effectively align large language models(LLMs) to human preferences. In this work, we propose Preference-Guided Reflective Sampling (PRS) Unlike random sampling, PRS employs a tree-based generation framework to enable more efficient sampling. PRS shows strong performance when applied in iterative offline RL training.
  arXiv  Detail & Related papers  (2024-08-22T07:18:46Z)
• SAIL: Self-Improving Efficient Online Alignment of Large Language Models [56.59644677997827]
  Reinforcement Learning from Human Feedback is a key method for aligning large language models with human preferences. Recent literature has focused on designing online RLHF methods but still lacks a unified conceptual formulation. Our approach significantly improves alignment performance on open-sourced datasets with minimal computational overhead.
  arXiv  Detail & Related papers  (2024-06-21T18:05:35Z)
• Aligning Large Language Models with Representation Editing: A Control Perspective [38.71496554018039]
  Fine-tuning large language models (LLMs) to align with human objectives is crucial for real-world applications. Test-time alignment techniques, such as prompting and guided decoding, do not modify the underlying model. We propose aligning LLMs through representation editing.
  arXiv  Detail & Related papers  (2024-06-10T01:21:31Z)
• Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts [95.09994361995389]
  Relative Preference Optimization (RPO) is designed to discern between more and less preferred responses derived from both identical and related prompts. RPO has demonstrated a superior ability to align large language models with user preferences and to improve their adaptability during the training process.
  arXiv  Detail & Related papers  (2024-02-12T22:47:57Z)
• Sample Efficient Preference Alignment in LLMs via Active Exploration [63.84454768573154]
  We take advantage of the fact that one can often choose contexts at which to obtain human feedback to most efficiently identify a good policy.<n>We propose an active exploration algorithm to efficiently select the data and provide theoretical proof that it has a worst-case regret bound.<n>Our method outperforms the baselines with limited samples of human preferences on several language models and four real-world datasets.
  arXiv  Detail & Related papers  (2023-12-01T00:54:02Z)
• Parameter-Efficient Tuning Helps Language Model Alignment [57.27390187540737]
  Previous works mainly adopt reinforcement learning (RLHF) and direct preference optimization (DPO) with human feedback for alignment. Controllable generation offers more flexibility with regard to data format. Our approach, alignMEnt with parameter-Efficient Tuning (MEET), improves the quality of control tokens.
  arXiv  Detail & Related papers  (2023-10-01T23:27:14Z)
• Read-only Prompt Optimization for Vision-Language Few-shot Learning [20.66798356082751]
  Learnable prompts can affect the internal representation within the self-attention module. We propose a novel approach, Read-only Prompt Optimization (RPO) Our experiments demonstrate that RPO outperforms CLIP and CoCoOp in base-to-new generalization and domain generalization.
  arXiv  Detail & Related papers  (2023-08-29T01:22:30Z)
• Reinforced Self-Training (ReST) for Language Modeling [56.75447441157628]
  Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call Reinforced Self-Training (ReST) Our results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.
  arXiv  Detail & Related papers  (2023-08-17T14:12:48Z)
• Towards Universal Sequence Representation Learning for Recommender Systems [98.02154164251846]
  We present a novel universal sequence representation learning approach, named UniSRec. The proposed approach utilizes the associated description text of items to learn transferable representations across different recommendation scenarios. Our approach can be effectively transferred to new recommendation domains or platforms in a parameter-efficient way.
  arXiv  Detail & Related papers  (2022-06-13T07:21:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.