More Than Accuracy: Towards Trustworthy Machine Learning Interfaces for Object Recognition

• URL: http://arxiv.org/abs/2008.01980v1
• Date: Wed, 5 Aug 2020 07:56:37 GMT
• Title: More Than Accuracy: Towards Trustworthy Machine Learning Interfaces for Object Recognition
• Authors: Hendrik Heuer, Andreas Breiter
• Abstract summary: This paper investigates the user experience of visualizations of a machine learning (ML) system that recognizes objects in images. We exposed users with a background in ML to three visualizations of three systems with different levels of accuracy. In interviews, we explored how the visualization helped users assess the accuracy of systems in use and how the visualization and the accuracy of the system affected trust and reliance.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper investigates the user experience of visualizations of a machine learning (ML) system that recognizes objects in images. This is important since even good systems can fail in unexpected ways as misclassifications on photo-sharing websites showed. In our study, we exposed users with a background in ML to three visualizations of three systems with different levels of accuracy. In interviews, we explored how the visualization helped users assess the accuracy of systems in use and how the visualization and the accuracy of the system affected trust and reliance. We found that participants do not only focus on accuracy when assessing ML systems. They also take the perceived plausibility and severity of misclassification into account and prefer seeing the probability of predictions. Semantically plausible errors are judged as less severe than errors that are implausible, which means that system accuracy could be communicated through the types of errors.

Related papers

• Uncertainty Estimation for 3D Object Detection via Evidential Learning [63.61283174146648]
  We introduce a framework for quantifying uncertainty in 3D object detection by leveraging an evidential learning loss on Bird's Eye View representations in the 3D detector. We demonstrate both the efficacy and importance of these uncertainty estimates on identifying out-of-distribution scenes, poorly localized objects, and missing (false negative) detections.
  arXiv  Detail & Related papers  (2024-10-31T13:13:32Z)
• A Systematic Review of Machine Learning Approaches for Detecting Deceptive Activities on Social Media: Methods, Challenges, and Biases [0.037693031068634524]
  This systematic review evaluates studies that apply machine learning (ML) and deep learning (DL) models to detect fake news, spam, and fake accounts on social media.
  arXiv  Detail & Related papers  (2024-10-26T23:55:50Z)
• Understanding How Blind Users Handle Object Recognition Errors: Strategies and Challenges [10.565823004989817]
  This paper presents a study aimed at understanding blind users' interaction with object recognition systems for identifying and avoiding errors. We conducted a user study involving 12 blind and low-vision participants. We gained insights into users' experiences, challenges, and strategies for identifying errors in camera-based assistive technologies and object recognition systems.
  arXiv  Detail & Related papers  (2024-08-06T17:09:56Z)
• Gaussian Mixture Models for Affordance Learning using Bayesian Networks [50.18477618198277]
  Affordances are fundamental descriptors of relationships between actions, objects and effects. This paper approaches the problem of an embodied agent exploring the world and learning these affordances autonomously from its sensory experiences.
  arXiv  Detail & Related papers  (2024-02-08T22:05:45Z)
• Real World Robustness from Systematic Noise [13.034436864136103]
  In this paper, we exhibit some long-neglected but frequent-happening adversarial examples caused by systematic error. To benchmark these real-world adversarial examples, we propose ImageNet-S dataset. For example, we find a normal ResNet-50 trained on ImageNet can have 1%-5% accuracy difference due to the systematic error.
  arXiv  Detail & Related papers  (2021-09-02T12:25:16Z)
• Multi Agent System for Machine Learning Under Uncertainty in Cyber Physical Manufacturing System [78.60415450507706]
  Recent advancements in predictive machine learning has led to its application in various use cases in manufacturing. Most research focused on maximising predictive accuracy without addressing the uncertainty associated with it. In this paper, we determine the sources of uncertainty in machine learning and establish the success criteria of a machine learning system to function well under uncertainty.
  arXiv  Detail & Related papers  (2021-07-28T10:28:05Z)
• Fairness Constraints in Semi-supervised Learning [56.48626493765908]
  We develop a framework for fair semi-supervised learning, which is formulated as an optimization problem. We theoretically analyze the source of discrimination in semi-supervised learning via bias, variance and noise decomposition. Our method is able to achieve fair semi-supervised learning, and reach a better trade-off between accuracy and fairness than fair supervised learning.
  arXiv  Detail & Related papers  (2020-09-14T04:25:59Z)
• Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy [8.11839312231511]
  Mixed-initiative systems allow users to interactively provide feedback to improve system performance. Our research investigates how the act of providing feedback can affect user understanding of an intelligent system and its accuracy.
  arXiv  Detail & Related papers  (2020-08-28T16:46:41Z)
• End-to-End Models for the Analysis of System 1 and System 2 Interactions based on Eye-Tracking Data [99.00520068425759]
  We propose a computational method, within a modified visual version of the well-known Stroop test, for the identification of different tasks and potential conflicts events. A statistical analysis shows that the selected variables can characterize the variation of attentive load within different scenarios. We show that Machine Learning techniques allow to distinguish between different tasks with a good classification accuracy.
  arXiv  Detail & Related papers  (2020-02-03T17:46:13Z)
• Training-free Monocular 3D Event Detection System for Traffic Surveillance [93.65240041833319]
  Existing event detection systems are mostly learning-based and have achieved convincing performance when a large amount of training data is available. In real-world scenarios, collecting sufficient labeled training data is expensive and sometimes impossible. We propose a training-free monocular 3D event detection system for traffic surveillance.
  arXiv  Detail & Related papers  (2020-02-01T04:42:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.