Personalization in Human-AI Teams: Improving the Compatibility-Accuracy Tradeoff

• URL: http://arxiv.org/abs/2004.02289v2
• Date: Wed, 19 Aug 2020 13:13:22 GMT
• Title: Personalization in Human-AI Teams: Improving the Compatibility-Accuracy Tradeoff
• Authors: Jonathan Martinez (1), Kobi Gal (1 and 2), Ece Kamar (3), Levi H. S. Lelis (4) ((1) Ben-Gurion University, (2) University of Edinburgh, (3) Microsoft Research, (4) University of Alberta)
• Abstract summary: We study the trade-off between improving the system's accuracy following an update and the compatibility of the updated system with prior user experience. We show that by personalizing the loss function to specific users, in some cases it is possible to improve the compatibility-accuracy trade-off with respect to these users.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI systems that model and interact with users can update their models over time to reflect new information and changes in the environment. Although these updates may improve the overall performance of the AI system, they may actually hurt the performance with respect to individual users. Prior work has studied the trade-off between improving the system's accuracy following an update and the compatibility of the updated system with prior user experience. The more the model is forced to be compatible with a prior version, the higher loss in accuracy it will incur. In this paper, we show that by personalizing the loss function to specific users, in some cases it is possible to improve the compatibility-accuracy trade-off with respect to these users (increase the compatibility of the model while sacrificing less accuracy). We present experimental results indicating that this approach provides moderate improvements on average (around 20%) but large improvements for certain users (up to 300%).

Related papers

• Collaborative-Enhanced Prediction of Spending on Newly Downloaded Mobile Games under Consumption Uncertainty [49.431361908465036]
  We propose a robust model training and evaluation framework to mitigate label variance and extremes. Within this framework, we introduce a collaborative-enhanced model designed to predict user game spending without relying on user IDs. Our approach demonstrates notable improvements over production models, achieving a remarkable textbf17.11% enhancement on offline data.
  arXiv  Detail & Related papers  (2024-04-12T07:47:02Z)
• Robustness-Congruent Adversarial Training for Secure Machine Learning Model Updates [13.911586916369108]
  We show that misclassifications in machine-learning models can affect robustness to adversarial examples. We propose a technique, named robustness-congruent adversarial training, to address this issue. We show that our algorithm and, more generally, learning with non-regression constraints, provides a theoretically-grounded framework to train consistent estimators.
  arXiv  Detail & Related papers  (2024-02-27T10:37:13Z)
• User Friendly and Adaptable Discriminative AI: Using the Lessons from the Success of LLMs and Image Generation Models [0.6926105253992517]
  We develop a new system architecture that enables users to work with discriminative models. Our approach has implications on improving trust, user-friendliness, and adaptability of these versatile but traditional prediction models.
  arXiv  Detail & Related papers  (2023-12-11T20:37:58Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• A Diachronic Perspective on User Trust in AI under Uncertainty [52.44939679369428]
  Modern NLP systems are often uncalibrated, resulting in confidently incorrect predictions that undermine user trust. We study the evolution of user trust in response to trust-eroding events using a betting game.
  arXiv  Detail & Related papers  (2023-10-20T14:41:46Z)
• Updating Clinical Risk Stratification Models Using Rank-Based Compatibility: Approaches for Evaluating and Optimizing Clinician-Model Team Performance [11.31203699519559]
  We propose a rank-based compatibility measure, $CR$, and a new loss function that aims to optimize discriminative performance while encouraging good compatibility. This work provides new tools to analyze and update risk stratification models used in clinical care.
  arXiv  Detail & Related papers  (2023-08-10T15:08:13Z)
• Improved Bayes Risk Can Yield Reduced Social Welfare Under Competition [99.7047087527422]
  In this work, we demonstrate that competition can fundamentally alter the behavior of machine learning scaling trends. We find many settings where improving data representation quality decreases the overall predictive accuracy across users. At a conceptual level, our work suggests that favorable scaling trends for individual model-providers need not translate to downstream improvements in social welfare.
  arXiv  Detail & Related papers  (2023-06-26T13:06:34Z)
• Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy [8.11839312231511]
  Mixed-initiative systems allow users to interactively provide feedback to improve system performance. Our research investigates how the act of providing feedback can affect user understanding of an intelligent system and its accuracy.
  arXiv  Detail & Related papers  (2020-08-28T16:46:41Z)
• An Empirical Analysis of Backward Compatibility in Machine Learning Systems [47.04803977692586]
  We consider how updates, intended to improve ML models, can introduce new errors that can significantly affect downstream systems and users. For example, updates in models used in cloud-based classification services, such as image recognition, can cause unexpected erroneous behavior.
  arXiv  Detail & Related papers  (2020-08-11T08:10:58Z)
• Optimizing Interactive Systems via Data-Driven Objectives [70.3578528542663]
  We propose an approach that infers the objective directly from observed user interactions. These inferences can be made regardless of prior knowledge and across different types of user behavior. We introduce Interactive System (ISO), a novel algorithm that uses these inferred objectives for optimization.
  arXiv  Detail & Related papers  (2020-06-19T20:49:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.