Aligning Model Properties via Conformal Risk Control

• URL: http://arxiv.org/abs/2406.18777v2
• Date: Tue, 05 Nov 2024 07:17:54 GMT
• Title: Aligning Model Properties via Conformal Risk Control
• Authors: William Overman, Jacqueline Jil Vallon, Mohsen Bayati,
• Abstract summary: Post-training alignment via human feedback shows promise, but is often limited to generative AI settings. In traditional non-generative settings with numerical or categorical outputs, detecting misalignment through single-sample outputs remains challenging. We propose interpreting model alignment through property testing, defining an aligned model $f$ as one belonging to a subset $mathcalP$ of functions.
• Score: 4.710921988115686
• License:
• Abstract: AI model alignment is crucial due to inadvertent biases in training data and the underspecified machine learning pipeline, where models with excellent test metrics may not meet end-user requirements. While post-training alignment via human feedback shows promise, these methods are often limited to generative AI settings where humans can interpret and provide feedback on model outputs. In traditional non-generative settings with numerical or categorical outputs, detecting misalignment through single-sample outputs remains challenging, and enforcing alignment during training requires repeating costly training processes. In this paper we consider an alternative strategy. We propose interpreting model alignment through property testing, defining an aligned model $f$ as one belonging to a subset $\mathcal{P}$ of functions that exhibit specific desired behaviors. We focus on post-processing a pre-trained model $f$ to better align with $\mathcal{P}$ using conformal risk control. Specifically, we develop a general procedure for converting queries for testing a given property $\mathcal{P}$ to a collection of loss functions suitable for use in a conformal risk control algorithm. We prove a probabilistic guarantee that the resulting conformal interval around $f$ contains a function approximately satisfying $\mathcal{P}$. We exhibit applications of our methodology on a collection of supervised learning datasets for (shape-constrained) properties such as monotonicity and concavity. The general procedure is flexible and can be applied to a wide range of desired properties. Finally, we prove that pre-trained models will always require alignment techniques even as model sizes or training data increase, as long as the training data contains even small biases.

Related papers

• MUSO: Achieving Exact Machine Unlearning in Over-Parameterized Regimes [19.664090734076712]
  Machine unlearning (MU) makes a well-trained model behave as if it had never been trained on specific data. We propose an alternating optimization algorithm that unifies the tasks of unlearning and relabeling. The algorithm's effectiveness, confirmed through numerical experiments, highlights its superior performance in unlearning across various scenarios.
  arXiv  Detail & Related papers  (2024-10-11T06:17:17Z)
• Adding Conditional Control to Diffusion Models with Reinforcement Learning [59.295203871547336]
  Diffusion models are powerful generative models that allow for precise control over the characteristics of the generated samples. This work presents a novel method based on reinforcement learning (RL) to add additional controls, leveraging an offline dataset.
  arXiv  Detail & Related papers  (2024-06-17T22:00:26Z)
• Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs [27.314239745883967]
  Training data attribution (TDA) methods trace a model's prediction on any given example back to specific influential training examples. We propose Simfluence, a new paradigm for TDA where the goal is not to produce a single influence score per example, but instead a training run simulator. Simfluence captures non-additive interactions and is often able to predict the spiky trajectory of individual example losses with surprising fidelity.
  arXiv  Detail & Related papers  (2023-03-14T17:47:25Z)
• On the Provable Advantage of Unsupervised Pretraining [26.065736182939222]
  Unsupervised pretraining is a critical component of modern large-scale machine learning systems. This paper studies a generic framework, where the unsupervised representation learning task is specified by an abstract class of latent variable models. Under a mild ''informative'' condition, our algorithm achieves an excess risk of $tildemathcalO(sqrtmathcalC_Phi/m + sqrtmathcalC_Psi/n)$ for downstream tasks.
  arXiv  Detail & Related papers  (2023-03-02T20:42:05Z)
• CAFA: Class-Aware Feature Alignment for Test-Time Adaptation [50.26963784271912]
  Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time. We propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously encourages a model to learn target representations in a class-discriminative manner.
  arXiv  Detail & Related papers  (2022-06-01T03:02:07Z)
• Conformal prediction for the design problem [72.14982816083297]
  In many real-world deployments of machine learning, we use a prediction algorithm to choose what data to test next. In such settings, there is a distinct type of distribution shift between the training and test data. We introduce a method to quantify predictive uncertainty in such settings.
  arXiv  Detail & Related papers  (2022-02-08T02:59:12Z)
• Datamodels: Predicting Predictions from Training Data [86.66720175866415]
  We present a conceptual framework, datamodeling, for analyzing the behavior of a model class in terms of the training data. We show that even simple linear datamodels can successfully predict model outputs.
  arXiv  Detail & Related papers  (2022-02-01T18:15:24Z)
• Generative Temporal Difference Learning for Infinite-Horizon Prediction [101.59882753763888]
  We introduce the $gamma$-model, a predictive model of environment dynamics with an infinite probabilistic horizon. We discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors.
  arXiv  Detail & Related papers  (2020-10-27T17:54:12Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.