Assisting Human Decisions in Document Matching

• URL: http://arxiv.org/abs/2302.08450v1
• Date: Thu, 16 Feb 2023 17:45:20 GMT
• Title: Assisting Human Decisions in Document Matching
• Authors: Joon Sik Kim, Valerie Chen, Danish Pruthi, Nihar B. Shah, Ameet Talwalkar
• Abstract summary: We devise a proxy matching task that allows us to evaluate which kinds of assistive information improve decision makers' performance. We find that providing black-box model explanations reduces users' accuracy on the matching task. On the other hand, custom methods that are designed to closely attend to some task-specific desiderata are found to be effective in improving user performance.
• Score: 52.79491990823573
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many practical applications, ranging from paper-reviewer assignment in peer review to job-applicant matching for hiring, require human decision makers to identify relevant matches by combining their expertise with predictions from machine learning models. In many such model-assisted document matching tasks, the decision makers have stressed the need for assistive information about the model outputs (or the data) to facilitate their decisions. In this paper, we devise a proxy matching task that allows us to evaluate which kinds of assistive information improve decision makers' performance (in terms of accuracy and time). Through a crowdsourced (N=271 participants) study, we find that providing black-box model explanations reduces users' accuracy on the matching task, contrary to the commonly-held belief that they can be helpful by allowing better understanding of the model. On the other hand, custom methods that are designed to closely attend to some task-specific desiderata are found to be effective in improving user performance. Surprisingly, we also find that the users' perceived utility of assistive information is misaligned with their objective utility (measured through their task performance).

Related papers

• Likelihood as a Performance Gauge for Retrieval-Augmented Generation [78.28197013467157]
  We show that likelihoods serve as an effective gauge for language model performance. We propose two methods that use question likelihood as a gauge for selecting and constructing prompts that lead to better performance.
  arXiv  Detail & Related papers  (2024-11-12T13:14:09Z)
• Decision-Aware Predictive Model Selection for Workforce Allocation [0.27309692684728615]
  We introduce a novel framework that utilizes machine learning to predict worker behavior. In our approach, the optimal predictive model used to represent a worker's behavior is determined by how that worker is allocated. We present a decision-aware optimization framework that integrates predictive model selection with worker allocation.
  arXiv  Detail & Related papers  (2024-10-10T13:59:43Z)
• Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents [110.25679611755962]
  Current language model-driven agents often lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions. We introduce Intention-in-Interaction (IN3), a novel benchmark designed to inspect users' implicit intentions through explicit queries. We empirically train Mistral-Interact, a powerful model that proactively assesses task vagueness, inquires user intentions, and refines them into actionable goals.
  arXiv  Detail & Related papers  (2024-02-14T14:36:30Z)
• Evaluating the Utility of Model Explanations for Model Development [54.23538543168767]
  We evaluate whether explanations can improve human decision-making in practical scenarios of machine learning model development. To our surprise, we did not find evidence of significant improvement on tasks when users were provided with any of the saliency maps. These findings suggest caution regarding the usefulness and potential for misunderstanding in saliency-based explanations.
  arXiv  Detail & Related papers  (2023-12-10T23:13:23Z)
• Optimising Human-AI Collaboration by Learning Convincing Explanations [62.81395661556852]
  We propose a method for a collaborative system that remains safe by having a human making decisions. Ardent enables efficient and effective decision-making by adapting to individual preferences for explanations.
  arXiv  Detail & Related papers  (2023-11-13T16:00:16Z)
• Creating user stereotypes for persona development from qualitative data through semi-automatic subspace clustering [0.0]
  We propose a method that employs the modelling of user stereotypes to automate part of the persona creation process. Results show that manual techniques differ between human persona designers leading to different results. The proposed algorithm provides similar results based on parameter input, but was more rigorous and will find optimal clusters.
  arXiv  Detail & Related papers  (2023-06-26T09:49:51Z)
• Designing Closed-Loop Models for Task Allocation [36.04165658325371]
  We exploit weak prior information on human-task similarity to bootstrap model training. We show that the use of such a weak prior can improve task allocation accuracy, even when human decision-makers are fallible and biased.
  arXiv  Detail & Related papers  (2023-05-31T13:57:56Z)
• Personalized Privacy Auditing and Optimization at Test Time [44.15285550981899]
  This paper asks whether it is necessary to require emphall input features for a model to return accurate predictions at test time. Under a personalized setting, each individual may need to release only a small subset of these features without impacting the final decisions. Evaluation over several learning tasks shows that individuals may be able to report as little as 10% of their information to ensure the same level of accuracy.
  arXiv  Detail & Related papers  (2023-01-31T20:16:59Z)
• Explainable Predictive Process Monitoring: A User Evaluation [62.41400549499849]
  Explainability is motivated by the lack of transparency of black-box Machine Learning approaches. We carry on a user evaluation on explanation approaches for Predictive Process Monitoring.
  arXiv  Detail & Related papers  (2022-02-15T22:24:21Z)
• DECE: Decision Explorer with Counterfactual Explanations for Machine Learning Models [36.50754934147469]
  We exploit the potential of counterfactual explanations to understand and explore the behavior of machine learning models. We design DECE, an interactive visualization system that helps understand and explore a model's decisions on individual instances and data subsets.
  arXiv  Detail & Related papers  (2020-08-19T09:44:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.