Uncertainty-aware Evaluation of Time-Series Classification for Online Handwriting Recognition with Domain Shift

• URL: http://arxiv.org/abs/2206.08640v1
• Date: Fri, 17 Jun 2022 09:05:01 GMT
• Title: Uncertainty-aware Evaluation of Time-Series Classification for Online Handwriting Recognition with Domain Shift
• Authors: Andreas Kla{\ss} and Sven M. Lorenz and Martin W. Lauer-Schmaltz and David R\"ugamer and Bernd Bischl and Christopher Mutschler and Felix Ott
• Abstract summary: In this paper, we focus on models for online handwriting recognition. The data is observed from a sensor-enhanced pen with the goal to write characters. Next to a better understanding of the model, UQ techniques can detect out-of-distribution of data.
• Score: 2.7015517125109247
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For many applications, analyzing the uncertainty of a machine learning model is indispensable. While research of uncertainty quantification (UQ) techniques is very advanced for computer vision applications, UQ methods for spatio-temporal data are less studied. In this paper, we focus on models for online handwriting recognition, one particular type of spatio-temporal data. The data is observed from a sensor-enhanced pen with the goal to classify written characters. We conduct a broad evaluation of aleatoric (data) and epistemic (model) UQ based on two prominent techniques for Bayesian inference, Stochastic Weight Averaging-Gaussian (SWAG) and Deep Ensembles. Next to a better understanding of the model, UQ techniques can detect out-of-distribution data and domain shifts when combining right-handed and left-handed writers (an underrepresented group).

Related papers

• Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph [83.90988015005934]
  Uncertainty quantification (UQ) is a critical component of machine learning (ML) applications. We introduce a novel benchmark that implements a collection of state-of-the-art UQ baselines. We conduct a large-scale empirical investigation of UQ and normalization techniques across nine tasks, and identify the most promising approaches.
  arXiv  Detail & Related papers  (2024-06-21T20:06:31Z)
• ScatterUQ: Interactive Uncertainty Visualizations for Multiclass Deep Learning Problems [0.0]
  ScatterUQ is an interactive system that provides targeted visualizations to allow users to better understand model performance in context-driven uncertainty settings. We demonstrate the effectiveness of ScatterUQ to explain model uncertainty for a multiclass image classification on a distance-aware neural network trained on Fashion-MNIST. Our results indicate that the ScatterUQ system should scale to arbitrary, multiclass datasets.
  arXiv  Detail & Related papers  (2023-08-08T21:17:03Z)
• Vector Quantized Wasserstein Auto-Encoder [57.29764749855623]
  We study learning deep discrete representations from the generative viewpoint. We endow discrete distributions over sequences of codewords and learn a deterministic decoder that transports the distribution over the sequences of codewords to the data distribution. We develop further theories to connect it with the clustering viewpoint of WS distance, allowing us to have a better and more controllable clustering solution.
  arXiv  Detail & Related papers  (2023-02-12T13:51:36Z)
• Utilizing Background Knowledge for Robust Reasoning over Traffic Situations [63.45021731775964]
  We focus on a complementary research aspect of Intelligent Transportation: traffic understanding. We scope our study to text-based methods and datasets given the abundant commonsense knowledge. We adopt three knowledge-driven approaches for zero-shot QA over traffic situations.
  arXiv  Detail & Related papers  (2022-12-04T09:17:24Z)
• A didactic approach to quantum machine learning with a single qubit [68.8204255655161]
  We focus on the case of learning with a single qubit, using data re-uploading techniques. We implement the different proposed formulations in toy and real-world datasets using the qiskit quantum computing SDK.
  arXiv  Detail & Related papers  (2022-11-23T18:25:32Z)
• TimeREISE: Time-series Randomized Evolving Input Sample Explanation [5.557646286040063]
  TimeREISE is a model attribution method specifically aligned to success in the context of time series classification. The method shows superior performance compared to existing approaches concerning different well-established measurements.
  arXiv  Detail & Related papers  (2022-02-16T09:40:13Z)
• Bayesian graph convolutional neural networks via tempered MCMC [0.41998444721319217]
  Deep learning models, such as convolutional neural networks, have long been applied to image and multi-media tasks. More recently, there has been more attention to unstructured data that can be represented via graphs. These types of data are often found in health and medicine, social networks, and research data repositories.
  arXiv  Detail & Related papers  (2021-04-17T04:03:25Z)
• Learning summary features of time series for likelihood free inference [93.08098361687722]
  We present a data-driven strategy for automatically learning summary features from time series data. Our results indicate that learning summary features from data can compete and even outperform LFI methods based on hand-crafted values.
  arXiv  Detail & Related papers  (2020-12-04T19:21:37Z)
• Neural Language Modeling for Contextualized Temporal Graph Generation [49.21890450444187]
  This paper presents the first study on using large-scale pre-trained language models for automated generation of an event-level temporal graph for a document.
  arXiv  Detail & Related papers  (2020-10-20T07:08:00Z)
• Uncertainty Quantification Using Neural Networks for Molecular Property Prediction [33.34534208450156]
  We systematically evaluate several methods on five benchmark datasets using multiple complementary performance metrics. None of the methods we tested is unequivocally superior to all others, and none produces a particularly reliable ranking of errors across multiple datasets. We conclude with a practical recommendation as to which existing techniques seem to perform well relative to others.
  arXiv  Detail & Related papers  (2020-05-20T13:31:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.