Two4Two: Evaluating Interpretable Machine Learning - A Synthetic Dataset For Controlled Experiments

• URL: http://arxiv.org/abs/2105.02825v1
• Date: Thu, 6 May 2021 17:14:39 GMT
• Title: Two4Two: Evaluating Interpretable Machine Learning - A Synthetic Dataset For Controlled Experiments
• Authors: Martin Schuessler, Philipp Wei{\ss}, Leon Sixt
• Abstract summary: We present a library that generates synthetic image data of two 3D abstract animals. The resulting data is suitable for algorithmic as well as human-subject evaluations. Our approach significantly lowers the barrier for conducting human subject evaluations.
• Score: 6.123324869194195
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A growing number of approaches exist to generate explanations for image classification. However, few of these approaches are subjected to human-subject evaluations, partly because it is challenging to design controlled experiments with natural image datasets, as they leave essential factors out of the researcher's control. With our approach, researchers can describe their desired dataset with only a few parameters. Based on these, our library generates synthetic image data of two 3D abstract animals. The resulting data is suitable for algorithmic as well as human-subject evaluations. Our user study results demonstrate that our method can create biases predictive enough for a classifier and subtle enough to be noticeable only to every second participant inspecting the data visually. Our approach significantly lowers the barrier for conducting human subject evaluations, thereby facilitating more rigorous investigations into interpretable machine learning. For our library and datasets see, https://github.com/mschuessler/two4two/

Related papers

• ConceptDrift: Uncovering Biases through the Lens of Foundation Models [5.025665239455297]
  We show that ConceptDrift can be scaled to automatically identify biases in datasets without human prior knowledge. We propose two bias identification evaluation protocols to fill the gap in the previous work and show that our method significantly improves over SoTA methods. Our method is not bounded to a single modality, and we empirically validate it both on image (Waterbirds, CelebA, ImageNet, and text datasets (CivilComments))
  arXiv  Detail & Related papers  (2024-10-24T17:59:16Z)
• Image change detection with only a few samples [7.5780621370948635]
  A major impediment of image change detection task is the lack of large annotated datasets covering a wide variety of scenes. We propose using simple image processing methods for generating synthetic but informative datasets. We then design an early fusion network based on object detection which could outperform the siamese neural network.
  arXiv  Detail & Related papers  (2023-11-07T07:01:35Z)
• Linking data separation, visual separation, and classifier performance using pseudo-labeling by contrastive learning [125.99533416395765]
  We argue that the performance of the final classifier depends on the data separation present in the latent space and visual separation present in the projection. We demonstrate our results by the classification of five real-world challenging image datasets of human intestinal parasites with only 1% supervised samples.
  arXiv  Detail & Related papers  (2023-02-06T10:01:38Z)
• BeCAPTCHA-Type: Biometric Keystroke Data Generation for Improved Bot Detection [63.447493500066045]
  This work proposes a data driven learning model for the synthesis of keystroke biometric data. The proposed method is compared with two statistical approaches based on Universal and User-dependent models. Our experimental framework considers a dataset with 136 million keystroke events from 168 thousand subjects.
  arXiv  Detail & Related papers  (2022-07-27T09:26:15Z)
• Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
  We propose methods to facilitate identification of training data artifacts. We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data. We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
  arXiv  Detail & Related papers  (2021-07-01T09:26:13Z)
• Tensor feature hallucination for few-shot learning [17.381648488344222]
  Few-shot classification addresses the challenge of classifying examples given limited supervision and limited data. Previous works on synthetic data generation for few-shot classification focus on exploiting complex models. We investigate how a simple and straightforward synthetic data generation method can be used effectively.
  arXiv  Detail & Related papers  (2021-06-09T18:25:08Z)
• DRG: Dual Relation Graph for Human-Object Interaction Detection [65.50707710054141]
  We tackle the challenging problem of human-object interaction (HOI) detection. Existing methods either recognize the interaction of each human-object pair in isolation or perform joint inference based on complex appearance-based features. In this paper, we leverage an abstract spatial-semantic representation to describe each human-object pair and aggregate the contextual information of the scene via a dual relation graph.
  arXiv  Detail & Related papers  (2020-08-26T17:59:40Z)
• Semi-Automatic Data Annotation guided by Feature Space Projection [117.9296191012968]
  We present a semi-automatic data annotation approach based on suitable feature space projection and semi-supervised label estimation. We validate our method on the popular MNIST dataset and on images of human intestinal parasites with and without fecal impurities. Our results demonstrate the added-value of visual analytics tools that combine complementary abilities of humans and machines for more effective machine learning.
  arXiv  Detail & Related papers  (2020-07-27T17:03:50Z)
• Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
  We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training. We experimentally verify that the new dataset can significantly improve the ability of the learned FER model. To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
  arXiv  Detail & Related papers  (2020-05-18T09:36:51Z)
• Do Saliency Models Detect Odd-One-Out Targets? New Datasets and Evaluations [15.374430656911498]
  We investigate singleton detection, which can be thought of as a canonical example of salience. We show that nearly all saliency algorithms do not adequately respond to singleton targets in synthetic and natural images.
  arXiv  Detail & Related papers  (2020-05-13T20:59:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.