Reward Generalization in RLHF: A Topological Perspective

• URL: http://arxiv.org/abs/2402.10184v7
• Date: Wed, 28 May 2025 11:59:41 GMT
• Title: Reward Generalization in RLHF: A Topological Perspective
• Authors: Tianyi Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Yang Han, Josef Dai, Xuehai Pan, Yaodong Yang,
• Abstract summary: We introduce a theory of reward generalization in reinforcement learning from human feedback.<n>At the macro level, we portray the RLHF information flow as an autoencoding process over behavior distributions.<n>At the micro level, we present induced Bayesian networks to model the impact of dataset topologies on reward generalization.
• Score: 11.860984070348348
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing alignment methods share a common topology of information flow, where reward information is collected from humans, modeled with preference learning, and used to tune language models. However, this shared topology has not been systematically characterized, nor have its alternatives been thoroughly explored, leaving the problems of low data efficiency and unreliable generalization unaddressed. As a solution, we introduce a theory of reward generalization in reinforcement learning from human feedback (RLHF), focusing on the topology of information flow at both macro and micro levels. At the macro level, we portray the RLHF information flow as an autoencoding process over behavior distributions, formalizing the RLHF objective of distributional consistency between human preference and model behavior. At the micro level, we present induced Bayesian networks to model the impact of dataset topologies on reward generalization. Combining analysis on both levels, we propose reward modeling from tree-structured preference information. It is shown to reduce reward uncertainty by up to $\Theta(\log n/\log\log n)$ times compared to baselines, where $n$ is the dataset size. Validation on three NLP tasks shows that it achieves an average win rate of 65% against baselines, thus improving reward generalization for free via topology design, while reducing the amount of data requiring annotation.

Related papers

• Sharpness-aware Federated Graph Learning [16.148982247077157]
  One of many impediments to applying graph neural networks (GNNs) to large-scale real-world graph data is the challenge of centralized training.<n> Federated graph learning (FGL) addresses this by enabling collaborative GNN model training without sharing private data.
  arXiv  Detail & Related papers  (2025-12-18T06:57:13Z)
• Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking [78.69179041551014]
  We propose an information-theoretic reward modeling framework based on the Information Bottleneck principle.<n>We show that InfoRM filters out preference-irrelevant information to alleviate reward misgeneralization.<n>We also introduce IBL, a distribution-level regularization that penalizes such deviations, effectively expanding the optimization landscape.
  arXiv  Detail & Related papers  (2025-10-15T15:51:59Z)
• Glocal Information Bottleneck for Time Series Imputation [70.41814118117311]
  Time Series Imputation aims to recover missing values in temporal data.<n>Existing models typically optimize the point-wise reconstruction loss, focusing on recovering numerical values (local information)<n>We propose a new training paradigm, Glocal Information Bottleneck (Glocal-IB)
  arXiv  Detail & Related papers  (2025-10-06T15:24:44Z)
• Alignment as Distribution Learning: Your Preference Model is Explicitly a Language Model [12.063078727764045]
  We argue that alignment via reinforcement learning from human feedback lacks theoretical justification and incentivizes deterministic solutions.<n>We propose three principled learning objectives: preference maximum likelihood estimation, preference distillation, and reverse KL minimization.<n>We empirically demonstrate that our distribution learning framework, especially preference distillation, consistently outperforms or matches the performances of RLHF and DPO.
  arXiv  Detail & Related papers  (2025-06-02T10:36:31Z)
• Generalization Capability for Imitation Learning [1.30536490219656]
  Imitation learning holds the promise of equipping robots with versatile skills by learning from expert demonstrations.<n>However, policies trained on finite datasets often struggle to generalize beyond the training distribution.<n>We present a unified perspective on the generalization capability of imitation learning, grounded in both information theorey and data distribution property.
  arXiv  Detail & Related papers  (2025-04-25T17:59:59Z)
• M$^3$-Impute: Mask-guided Representation Learning for Missing Value Imputation [12.174699459648842]
  M$3$-Impute aims to explicitly leverage the missingness information and such correlations with novel masking schemes. Experiment results show the effectiveness of M$3$-Impute by achieving 20 best and 4 second-best MAE scores on average.
  arXiv  Detail & Related papers  (2024-10-11T13:25:32Z)
• Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning? [50.03434441234569]
  Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention.
  arXiv  Detail & Related papers  (2024-09-05T19:00:18Z)
• BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learning [39.090104460303415]
  offline model-based reinforcement learning (MBRL) enhances data efficiency by utilizing pre-collected datasets to learn models and policies.<n>This paper first identifies the primary source of this mismatch comes from the underlying confounders present in offline data.<n>We introduce textbfBilintextbfEar textbfCAUSal rtextbfEpresentation(BECAUSE), an algorithm to capture causal representation for both states.
  arXiv  Detail & Related papers  (2024-07-15T17:59:23Z)
• Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis [16.288866201806382]
  We develop a model-free RLHF best policy identification algorithm, called $mathsfBSAD$, without explicit reward model inference. The algorithm identifies the optimal policy directly from human preference information in a backward manner.
  arXiv  Detail & Related papers  (2024-06-11T17:01:41Z)
• Graph Out-of-Distribution Generalization via Causal Intervention [69.70137479660113]
  We introduce a conceptually simple yet principled approach for training robust graph neural networks (GNNs) under node-level distribution shifts. Our method resorts to a new learning objective derived from causal inference that coordinates an environment estimator and a mixture-of-expert GNN predictor. Our model can effectively enhance generalization with various types of distribution shifts and yield up to 27.4% accuracy improvement over state-of-the-arts on graph OOD generalization benchmarks.
  arXiv  Detail & Related papers  (2024-02-18T07:49:22Z)
• InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling [66.3072381478251]
  Reward hacking, also termed reward overoptimization, remains a critical challenge. We propose a framework for reward modeling, namely InfoRM, by introducing a variational information bottleneck objective. We show that InfoRM's overoptimization detection mechanism is not only effective but also robust across a broad range of datasets.
  arXiv  Detail & Related papers  (2024-02-14T17:49:07Z)
• Out of the Ordinary: Spectrally Adapting Regression for Covariate Shift [12.770658031721435]
  We propose a method for adapting the weights of the last layer of a pre-trained neural regression model to perform better on input data originating from a different distribution. We demonstrate how this lightweight spectral adaptation procedure can improve out-of-distribution performance for synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-12-29T04:15:58Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• Classification of Data Generated by Gaussian Mixture Models Using Deep ReLU Networks [28.437011792990347]
  This paper studies the binary classification of data from $math RMs. generated under Gaussian Mixture networks. We obtain $d2013x neural analysis rates for the first time convergence rates. Results provide a theoretical verification of deep neural networks in practical classification problems.
  arXiv  Detail & Related papers  (2023-08-15T20:40:42Z)
• Topology-aware Robust Optimization for Out-of-distribution Generalization [18.436575017126323]
  Out-of-distribution (OOD) generalization is a challenging machine learning problem yet highly desirable in many high-stake applications. We propose topology-aware robust optimization (TRO) that seamlessly integrates distributional topology in a principled optimization framework. We theoretically demonstrate the effectiveness of our approach and empirically show that it significantly outperforms the state of the arts in a wide range of tasks including classification, regression, and semantic segmentation.
  arXiv  Detail & Related papers  (2023-07-26T03:48:37Z)
• Towards Understanding and Improving GFlowNet Training [71.85707593318297]
  We introduce an efficient evaluation strategy to compare the learned sampling distribution to the target reward distribution. We propose prioritized replay training of high-reward $x$, relative edge flow policy parametrization, and a novel guided trajectory balance objective.
  arXiv  Detail & Related papers  (2023-05-11T22:50:41Z)
• GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond [101.5329678997916]
  We study sample efficient reinforcement learning (RL) under the general framework of interactive decision making. We propose a novel complexity measure, generalized eluder coefficient (GEC), which characterizes the fundamental tradeoff between exploration and exploitation. We show that RL problems with low GEC form a remarkably rich class, which subsumes low Bellman eluder dimension problems, bilinear class, low witness rank problems, PO-bilinear class, and generalized regular PSR.
  arXiv  Detail & Related papers  (2022-11-03T16:42:40Z)
• Optimal Propagation for Graph Neural Networks [51.08426265813481]
  We propose a bi-level optimization approach for learning the optimal graph structure. We also explore a low-rank approximation model for further reducing the time complexity.
  arXiv  Detail & Related papers  (2022-05-06T03:37:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.