An Empirical Examination of the Evaluative AI Framework

• URL: http://arxiv.org/abs/2411.08583v1
• Date: Wed, 13 Nov 2024 13:03:49 GMT
• Title: An Empirical Examination of the Evaluative AI Framework
• Authors: Jaroslaw Kornowicz,
• Abstract summary: This study empirically examines the "Evaluative AI" framework, which aims to enhance the decision-making process for AI users. Rather than offering direct recommendations, this framework presents users pro and con evidence for hypotheses to support more informed decisions.
• Score: 0.0
• License:
• Abstract: This study empirically examines the "Evaluative AI" framework, which aims to enhance the decision-making process for AI users by transitioning from a recommendation-based approach to a hypothesis-driven one. Rather than offering direct recommendations, this framework presents users pro and con evidence for hypotheses to support more informed decisions. However, findings from the current behavioral experiment reveal no significant improvement in decision-making performance and limited user engagement with the evidence provided, resulting in cognitive processes similar to those observed in traditional AI systems. Despite these results, the framework still holds promise for further exploration in future research.

Related papers

• Negotiating the Shared Agency between Humans & AI in the Recommender System [1.4249472316161877]
  Concerns about user agency have arisen due to the inherent opacity (information asymmetry) and the nature of one-way output (power asymmetry) on algorithms. We seek to understand how types of agency impact user perception and experience, and bring empirical evidence to refine the guidelines and designs for human-AI interactive systems.
  arXiv  Detail & Related papers  (2024-03-23T19:23:08Z)
• Towards the New XAI: A Hypothesis-Driven Approach to Decision Support Using Evidence [9.916507773707917]
  We describe and evaluate an approach for hypothesis-driven AI based on the Weight of Evidence (WoE) framework. We show that our hypothesis-driven approach increases decision accuracy and reduces reliance compared to a recommendation-driven approach.
  arXiv  Detail & Related papers  (2024-02-02T10:28:24Z)
• Pioneering EEG Motor Imagery Classification Through Counterfactual Analysis [26.859082755430595]
  We introduce and explore a novel non-generative approach to counterfactual explanation (CE) This approach assesses the model's decision-making process by strategically swapping patches derived from time-frequency analyses. The empirical results serve not only to validate the efficacy of our proposed approach but also to reinforce human confidence in the model's predictive capabilities.
  arXiv  Detail & Related papers  (2023-11-10T08:22:09Z)
• The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice [64.29355073494125]
  This article aims to ground what we dub the "participatory turn" in AI design by synthesizing existing theoretical literature on participation. We articulate empirical findings concerning the current state of participatory practice in AI design based on an analysis of recently published research and semi-structured interviews with 12 AI researchers and practitioners.
  arXiv  Detail & Related papers  (2023-10-02T05:30:42Z)
• Rational Decision-Making Agent with Internalized Utility Judgment [91.80700126895927]
  Large language models (LLMs) have demonstrated remarkable advancements and have attracted significant efforts to develop LLMs into agents capable of executing intricate multi-step decision-making tasks beyond traditional NLP applications. This paper proposes RadAgent, which fosters the development of its rationality through an iterative framework involving Experience Exploration and Utility Learning. Experimental results on the ToolBench dataset demonstrate RadAgent's superiority over baselines, achieving over 10% improvement in Pass Rate on diverse tasks.
  arXiv  Detail & Related papers  (2023-08-24T03:11:45Z)
• E-NER: Evidential Deep Learning for Trustworthy Named Entity Recognition [69.87816981427858]
  Most named entity recognition (NER) systems focus on improving model performance, ignoring the need to quantify model uncertainty. Evidential deep learning (EDL) has recently been proposed as a promising solution to explicitly model predictive uncertainty for classification tasks. We propose a trustworthy NER framework named E-NER by introducing two uncertainty-guided loss terms to the conventional EDL, along with a series of uncertainty-guided training strategies.
  arXiv  Detail & Related papers  (2023-05-29T02:36:16Z)
• Debiasing Recommendation by Learning Identifiable Latent Confounders [49.16119112336605]
  Confounding bias arises due to the presence of unmeasured variables that can affect both a user's exposure and feedback. Existing methods either (1) make untenable assumptions about these unmeasured variables or (2) directly infer latent confounders from users' exposure. We propose a novel method, i.e., identifiable deconfounder (iDCF), which leverages a set of proxy variables to resolve the aforementioned non-identification issue.
  arXiv  Detail & Related papers  (2023-02-10T05:10:26Z)
• In Search of Insights, Not Magic Bullets: Towards Demystification of the Model Selection Dilemma in Heterogeneous Treatment Effect Estimation [92.51773744318119]
  This paper empirically investigates the strengths and weaknesses of different model selection criteria. We highlight that there is a complex interplay between selection strategies, candidate estimators and the data used for comparing them.
  arXiv  Detail & Related papers  (2023-02-06T16:55:37Z)
• Justification of Recommender Systems Results: A Service-based Approach [4.640835690336653]
  We propose a novel justification approach that uses service models to extract experience data from reviews concerning all the stages of interaction with items. In a user study, we compared our approach with baselines reflecting the state of the art in the justification of recommender systems results. Our models received higher Interface Adequacy and Satisfaction evaluations by users having different levels of Curiosity or low Need for Cognition (NfC) These findings encourage the adoption of service models to justify recommender systems results but suggest the investigation of personalization strategies to suit diverse interaction needs.
  arXiv  Detail & Related papers  (2022-11-07T11:08:19Z)
• Efficient Real-world Testing of Causal Decision Making via Bayesian Experimental Design for Contextual Optimisation [12.37745209793872]
  We introduce a model-agnostic framework for gathering data to evaluate and improve contextual decision making. Our method is used for the data-efficient evaluation of the regret of past treatment assignments.
  arXiv  Detail & Related papers  (2022-07-12T01:20:11Z)
• Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies [79.60322329952453]
  We show how to develop interpretable representations of how agents make decisions. By understanding the decision-making processes underlying a set of observed trajectories, we cast the policy inference problem as the inverse to this online learning problem. We introduce a practical algorithm for retrospectively estimating such perceived effects, alongside the process through which agents update them. Through application to the analysis of UNOS organ donation acceptance decisions, we demonstrate that our approach can bring valuable insights into the factors that govern decision processes and how they change over time.
  arXiv  Detail & Related papers  (2022-03-14T17:40:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.