Maintaining Stability and Plasticity for Predictive Churn Reduction
- URL: http://arxiv.org/abs/2305.04135v1
- Date: Sat, 6 May 2023 20:56:20 GMT
- Title: Maintaining Stability and Plasticity for Predictive Churn Reduction
- Authors: George Adam, Benjamin Haibe-Kains, Anna Goldenberg
- Abstract summary: We propose a solution called Accumulated Model Combination (AMC)
AMC is a general technique and we propose several instances of it, each having their own advantages depending on the model and data properties.
- Score: 8.971668467496055
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deployed machine learning models should be updated to take advantage of a
larger sample size to improve performance, as more data is gathered over time.
Unfortunately, even when model updates improve aggregate metrics such as
accuracy, they can lead to errors on samples that were correctly predicted by
the previous model causing per-sample regression in performance known as
predictive churn. Such prediction flips erode user trust thereby reducing the
effectiveness of the human-AI team as a whole. We propose a solution called
Accumulated Model Combination (AMC) based keeping the previous and current
model version, and generating a meta-output using the prediction of the two
models. AMC is a general technique and we propose several instances of it, each
having their own advantages depending on the model and data properties. AMC
requires minimal additional computation and changes to training procedures. We
motivate the need for AMC by showing the difficulty of making a single model
consistent with its own predictions throughout training thereby revealing an
implicit stability-plasticity tradeoff when training a single model. We
demonstrate the effectiveness of AMC on a variety of modalities including
computer vision, text, and tabular datasets comparing against state-of-the-art
churn reduction methods, and showing superior churn reduction ability compared
to all existing methods while being more efficient than ensembles.
Related papers
- On conditional diffusion models for PDE simulations [53.01911265639582]
We study score-based diffusion models for forecasting and assimilation of sparse observations.
We propose an autoregressive sampling approach that significantly improves performance in forecasting.
We also propose a new training strategy for conditional score-based models that achieves stable performance over a range of history lengths.
arXiv Detail & Related papers (2024-10-21T18:31:04Z) - Learning Augmentation Policies from A Model Zoo for Time Series Forecasting [58.66211334969299]
We introduce AutoTSAug, a learnable data augmentation method based on reinforcement learning.
By augmenting the marginal samples with a learnable policy, AutoTSAug substantially improves forecasting performance.
arXiv Detail & Related papers (2024-09-10T07:34:19Z) - Towards Stable Machine Learning Model Retraining via Slowly Varying Sequences [6.067007470552307]
We propose a methodology for finding sequences of machine learning models that are stable across retraining iterations.
We develop a mixed-integer optimization formulation that is guaranteed to recover optimal models.
Our method shows stronger stability than greedily trained models with a small, controllable sacrifice in predictive power.
arXiv Detail & Related papers (2024-03-28T22:45:38Z) - Consensus-Adaptive RANSAC [104.87576373187426]
We propose a new RANSAC framework that learns to explore the parameter space by considering the residuals seen so far via a novel attention layer.
The attention mechanism operates on a batch of point-to-model residuals, and updates a per-point estimation state to take into account the consensus found through a lightweight one-step transformer.
arXiv Detail & Related papers (2023-07-26T08:25:46Z) - A Comprehensive Modeling Approach for Crop Yield Forecasts using
AI-based Methods and Crop Simulation Models [0.21094707683348418]
We propose a comprehensive approach for yield forecasting that combines data-driven solutions, crop simulation models, and model surrogates.
Our data-driven modeling approach outperforms previous works with yield correlation predictions close to 91%.
arXiv Detail & Related papers (2023-06-16T18:13:24Z) - Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models.
This creates a barrier to fusing knowledge across individual models to yield a better single model.
We propose a dataless knowledge fusion method that merges models in their parameter space.
arXiv Detail & Related papers (2022-12-19T20:46:43Z) - End-to-End Weak Supervision [15.125993628007972]
We propose an end-to-end approach for directly learning the downstream model.
We show improved performance over prior work in terms of end model performance on downstream test sets.
arXiv Detail & Related papers (2021-07-05T19:10:11Z) - Test-time Collective Prediction [73.74982509510961]
Multiple parties in machine learning want to jointly make predictions on future test points.
Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters.
We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
arXiv Detail & Related papers (2021-06-22T18:29:58Z) - Contrastive Model Inversion for Data-Free Knowledge Distillation [60.08025054715192]
We propose Contrastive Model Inversion, where the data diversity is explicitly modeled as an optimizable objective.
Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination.
Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI achieves significantly superior performance when the generated data are used for knowledge distillation.
arXiv Detail & Related papers (2021-05-18T15:13:00Z) - Model-based Policy Optimization with Unsupervised Model Adaptation [37.09948645461043]
We investigate how to bridge the gap between real and simulated data due to inaccurate model estimation for better policy optimization.
We propose a novel model-based reinforcement learning framework AMPO, which introduces unsupervised model adaptation.
Our approach achieves state-of-the-art performance in terms of sample efficiency on a range of continuous control benchmark tasks.
arXiv Detail & Related papers (2020-10-19T14:19:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.