Metric Learning and Adaptive Boundary for Out-of-Domain Detection

• URL: http://arxiv.org/abs/2204.10849v1
• Date: Fri, 22 Apr 2022 17:54:55 GMT
• Title: Metric Learning and Adaptive Boundary for Out-of-Domain Detection
• Authors: Petr Lorenc, Tommaso Gargiani, Jan Pichl, Jakub Konr\'ad, Petr Marek, Ond\v{r}ej Kobza, Jan \v{S}ediv\'y
• Abstract summary: We have designed an OOD detection algorithm independent of OOD data. Our algorithm is based on a simple but efficient approach of combining metric learning with adaptive decision boundary. Compared to other algorithms, we have found that our proposed algorithm has significantly improved OOD performance in a scenario with a lower number of classes.
• Score: 0.9236074230806579
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conversational agents are usually designed for closed-world environments. Unfortunately, users can behave unexpectedly. Based on the open-world environment, we often encounter the situation that the training and test data are sampled from different distributions. Then, data from different distributions are called out-of-domain (OOD). A robust conversational agent needs to react to these OOD utterances adequately. Thus, the importance of robust OOD detection is emphasized. Unfortunately, collecting OOD data is a challenging task. We have designed an OOD detection algorithm independent of OOD data that outperforms a wide range of current state-of-the-art algorithms on publicly available datasets. Our algorithm is based on a simple but efficient approach of combining metric learning with adaptive decision boundary. Furthermore, compared to other algorithms, we have found that our proposed algorithm has significantly improved OOD performance in a scenario with a lower number of classes while preserving the accuracy for in-domain (IND) classes.

Related papers

• EAT: Towards Long-Tailed Out-of-Distribution Detection [55.380390767978554]
  This paper addresses the challenging task of long-tailed OOD detection. The main difficulty lies in distinguishing OOD data from samples belonging to the tail classes. We propose two simple ideas: (1) Expanding the in-distribution class space by introducing multiple abstention classes, and (2) Augmenting the context-limited tail classes by overlaying images onto the context-rich OOD data.
  arXiv  Detail & Related papers  (2023-12-14T13:47:13Z)
• Beyond AUROC & co. for evaluating out-of-distribution detection performance [50.88341818412508]
  Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs. We propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples.
  arXiv  Detail & Related papers  (2023-06-26T12:51:32Z)
• Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data [7.221206118679026]
  We show that existing post-hoc methods perform quite differently compared to when evaluated only on OOD detection. We propose a novel method for SCOD, Softmax Information Retaining Combination (SIRC), that augments softmax-based confidence scores with feature-agnostic information. Experiments on a wide variety of ImageNet-scale datasets and convolutional neural network architectures show that SIRC is able to consistently match or outperform the baseline for SCOD.
  arXiv  Detail & Related papers  (2022-07-15T14:39:57Z)
• Igeood: An Information Geometry Approach to Out-of-Distribution Detection [35.04325145919005]
  We introduce Igeood, an effective method for detecting out-of-distribution (OOD) samples. Igeood applies to any pre-trained neural network, works under various degrees of access to the machine learning model. We show that Igeood outperforms competing state-of-the-art methods on a variety of network architectures and datasets.
  arXiv  Detail & Related papers  (2022-03-15T11:26:35Z)
• Training OOD Detectors in their Natural Habitats [31.565635192716712]
  Out-of-distribution (OOD) detection is important for machine learning models deployed in the wild. Recent methods use auxiliary outlier data to regularize the model for improved OOD detection. We propose a novel framework that leverages wild mixture data -- that naturally consists of both ID and OOD samples.
  arXiv  Detail & Related papers  (2022-02-07T15:38:39Z)
• Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation [76.2621758731288]
  We tackle the detection of out-of-distribution (OOD) objects in semantic segmentation. Our main contribution is a new OOD detection architecture called ObsNet associated with a dedicated training scheme based on Local Adversarial Attacks (LAA) We show it obtains top performances both in speed and accuracy when compared to ten recent methods of the literature on three different datasets.
  arXiv  Detail & Related papers  (2021-08-03T17:09:56Z)
• Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLU [70.44344060176952]
  Intent classification is a major task in spoken language understanding (SLU) Recent works have shown that using extra data and labels can improve the OOD detection performance. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection.
  arXiv  Detail & Related papers  (2021-06-28T08:27:38Z)
• Can Active Learning Preemptively Mitigate Fairness Issues? [66.84854430781097]
  dataset bias is one of the prevailing causes of unfairness in machine learning. We study whether models trained with uncertainty-based ALs are fairer in their decisions with respect to a protected class. We also explore the interaction of algorithmic fairness methods such as gradient reversal (GRAD) and BALD.
  arXiv  Detail & Related papers  (2021-04-14T14:20:22Z)
• Probing Predictions on OOD Images via Nearest Categories [97.055916832257]
  We study out-of-distribution (OOD) prediction behavior of neural networks when they classify images from unseen classes or corrupted images. We introduce a new measure, nearest category generalization (NCG), where we compute the fraction of OOD inputs that are classified with the same label as their nearest neighbor in the training set. We find that robust networks have consistently higher NCG accuracy than natural training, even when the OOD data is much farther away than the robustness radius.
  arXiv  Detail & Related papers  (2020-11-17T07:42:27Z)
• Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection In Task Oriented Dialog [24.653367921046442]
  We focus on OOD detection for natural language sentence inputs to task-based dialog systems. We release a dataset of 4K OOD examples for the publicly available dataset fromSchuster et al.
  arXiv  Detail & Related papers  (2019-12-30T03:31:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.