A Method for Learning Value Systems in Generative AI

• URL: http://arxiv.org/abs/2607.16903v1
• Date: Sat, 18 Jul 2026 17:37:50 GMT
• Title: A Method for Learning Value Systems in Generative AI
• Abstract summary: This work addresses the lack of grounded value learning methods in generative AI.<n>We adapt a previously validated value system learning method to the generative AI setting.<n>Our algorithm dynamically prioritizes the grounding learning process.
• Score: 1.376408511310322
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Value-aware AI systems require explicit computational representations of human values (groundings) and their aggregation into value systems in order to align their decisions with ours. As such representations are difficult to elicit, value learning seeks to infer them by observing human behaviour. This work addresses the lack of grounded value learning methods in generative AI: existing approaches typically replicate human preferences without awareness of the multidimensional structure of value alignment, or lack principled value system elicitation methods. To address these gaps, we adapt a previously validated value system learning method to the generative AI setting, which, based on pairwise prompt-response preference data, simultaneously learns: i) an implementation of a grounding for a set of values given by a multi-objective reward model, and ii) a value system representation in the form of a weighted linear scalarization of the previous grounding model. To ensure that the learned value systems are based on coherent value representations, our algorithm dynamically prioritizes the grounding learning process. We evaluate the method against baselines and a contemporary method on prompt-response preference datasets. Results show competitive performance and minimal trade-offs against the baselines, while improving explainability.

Related papers

• Measuring the Machine: Evaluating Generative AI as Pluralist Sociotechical Systems [0.0]
  The thesis argues that generative AI must be evaluated as a pluralist sociotechnical system.<n>MaSH Loops is a framework for tracing how models, users, and institutions co-construct meaning and values.<n>The World Values Benchmark introduces a distributional approach grounded in World Values Survey data.
  arXiv  Detail & Related papers  (2026-04-22T13:29:33Z)
• Learning the Value Systems of Societies with Preference-based Multi-objective Reinforcement Learning [4.735670734773144]
  Value-aware AI should recognise human values and adapt to the value systems (value-based preferences) of different users.<n>We propose algorithms for learning models of value alignment and value systems for a society of agents.
  arXiv  Detail & Related papers  (2026-02-09T16:06:36Z)
• Learning the Value Systems of Agents with Preference-based and Inverse Reinforcement Learning [1.6970482663318245]
  Agreement Technologies refer to open computer systems in which autonomous software agents interact with one another.<n>We propose a novel method to automatically emphlearn value systems from observations and human demonstrations.
  arXiv  Detail & Related papers  (2026-02-04T13:07:15Z)
• A Survey on Generative Recommendation: Data, Model, and Tasks [55.36322811257545]
  generative recommendation reconceptualizes recommendation as a generation task rather than discriminative scoring.<n>This survey provides a comprehensive examination through a unified tripartite framework spanning data, model, and task dimensions.<n>We identify five key advantages: world knowledge integration, natural language understanding, reasoning capabilities, scaling laws, and creative generation.
  arXiv  Detail & Related papers  (2025-10-31T04:02:58Z)
• Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods [23.6050988823262]
  Machine Unlearning (MU) aims to remove target training data from a trained model so that the removed data no longer influences the model's behavior.<n>Yet, researchers in this rapidly emerging field face challenges in analyzing and understanding the behavior of different MU methods.<n>We introduce a visual analytics system, Unlearning Comparator, designed to facilitate the systematic evaluation of MU methods.
  arXiv  Detail & Related papers  (2025-08-18T08:53:53Z)
• Learning the Value Systems of Societies from Preferences [1.3836987591220347]
  Aligning AI systems with human values and the value-based preferences of various stakeholders is key in ethical AI.<n>In value-aware AI systems, decision-making draws upon explicit computational representations of individual values.<n>We propose a method to address the problem of learning the value systems of societies.
  arXiv  Detail & Related papers  (2025-07-28T11:25:55Z)
• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Heterogeneous Value Alignment Evaluation for Large Language Models [91.96728871418]
  Large Language Models (LLMs) have made it crucial to align their values with those of humans. We propose a Heterogeneous Value Alignment Evaluation (HVAE) system to assess the success of aligning LLMs with heterogeneous values.
  arXiv  Detail & Related papers  (2023-05-26T02:34:20Z)
• DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation [57.358212277226315]
  In imitation learning from observation IfO, a learning agent seeks to imitate a demonstrating agent using only observations of the demonstrated behavior without access to the control signals generated by the demonstrator. Recent methods based on adversarial imitation learning have led to state-of-the-art performance on IfO problems, but they typically suffer from high sample complexity due to a reliance on data-inefficient, model-free reinforcement learning algorithms. This issue makes them impractical to deploy in real-world settings, where gathering samples can incur high costs in terms of time, energy, and risk. We propose a more data-efficient IfO algorithm
  arXiv  Detail & Related papers  (2021-03-31T23:46:32Z)
• Self-organizing Democratized Learning: Towards Large-scale Distributed Learning Systems [71.14339738190202]
  democratized learning (Dem-AI) lays out a holistic philosophy with underlying principles for building large-scale distributed and democratized machine learning systems. Inspired by Dem-AI philosophy, a novel distributed learning approach is proposed in this paper. The proposed algorithms demonstrate better results in the generalization performance of learning models in agents compared to the conventional FL algorithms.
  arXiv  Detail & Related papers  (2020-07-07T08:34:48Z)
• Active Learning for Nonlinear System Identification with Guarantees [102.43355665393067]
  We study a class of nonlinear dynamical systems whose state transitions depend linearly on a known feature embedding of state-action pairs. We propose an active learning approach that achieves this by repeating three steps: trajectory planning, trajectory tracking, and re-estimation of the system from all available data. We show that our method estimates nonlinear dynamical systems at a parametric rate, similar to the statistical rate of standard linear regression.
  arXiv  Detail & Related papers  (2020-06-18T04:54:11Z)
• Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning [75.56839075060819]
  Traditional robotic approaches rely on an accurate model of the environment, a detailed description of how to perform the task, and a robust perception system to keep track of the current state. reinforcement learning approaches can operate directly from raw sensory inputs with only a reward signal to describe the task, but are extremely sample-inefficient and brittle. In this work, we combine the strengths of model-based methods with the flexibility of learning-based methods to obtain a general method that is able to overcome inaccuracies in the robotics perception/actuation pipeline.
  arXiv  Detail & Related papers  (2020-05-21T19:47:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.