DAPPER: Label-Free Performance Estimation after Personalization for Heterogeneous Mobile Sensing

• URL: http://arxiv.org/abs/2111.11053v2
• Date: Tue, 13 Jun 2023 17:10:22 GMT
• Title: DAPPER: Label-Free Performance Estimation after Personalization for Heterogeneous Mobile Sensing
• Authors: Taesik Gong, Yewon Kim, Adiba Orzikulova, Yunxin Liu, Sung Ju Hwang, Jinwoo Shin, Sung-Ju Lee
• Abstract summary: We present DAPPER (Domain AdaPtation Performance EstimatoR) that estimates the adaptation performance in a target domain with unlabeled target data. Our evaluation with four real-world sensing datasets compared against six baselines shows that DAPPER outperforms the state-of-the-art baseline by 39.8% in estimation accuracy.
• Score: 95.18236298557721
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many applications utilize sensors in mobile devices and machine learning to provide novel services. However, various factors such as different users, devices, and environments impact the performance of such applications, thus making the domain shift (i.e., distributional shift between the training domain and the target domain) a critical issue in mobile sensing. Despite attempts in domain adaptation to solve this challenging problem, their performance is unreliable due to the complex interplay among diverse factors. In principle, the performance uncertainty can be identified and redeemed by performance validation with ground-truth labels. However, it is infeasible for every user to collect high-quality, sufficient labeled data. To address the issue, we present DAPPER (Domain AdaPtation Performance EstimatoR) that estimates the adaptation performance in a target domain with only unlabeled target data. Our key idea is to approximate the model performance based on the mutual information between the model inputs and corresponding outputs. Our evaluation with four real-world sensing datasets compared against six baselines shows that on average, DAPPER outperforms the state-of-the-art baseline by 39.8% in estimation accuracy. Moreover, our on-device experiment shows that DAPPER achieves up to 396X less computation overhead compared with the baselines.

Related papers

• Self-Training with Pseudo-Label Scorer for Aspect Sentiment Quad Prediction [54.23208041792073]
  Aspect Sentiment Quad Prediction (ASQP) aims to predict all quads (aspect term, aspect category, opinion term, sentiment polarity) for a given review. A key challenge in the ASQP task is the scarcity of labeled data, which limits the performance of existing methods. We propose a self-training framework with a pseudo-label scorer, wherein a scorer assesses the match between reviews and their pseudo-labels.
  arXiv  Detail & Related papers  (2024-06-26T05:30:21Z)
• Sensitivity-Informed Augmentation for Robust Segmentation [21.609070498399863]
  Internal noises such as variations in camera quality or lens distortion can affect the performance of segmentation models. We present an efficient, adaptable, and gradient-free method to enhance the robustness of learning-based segmentation models across training.
  arXiv  Detail & Related papers  (2024-06-03T15:25:45Z)
• Test-time Assessment of a Model's Performance on Unseen Domains via Optimal Transport [8.425690424016986]
  Gauging the performance of ML models on data from unseen domains at test-time is essential. It is essential to develop metrics that can provide insights into the model's performance at test time. We propose a metric based on Optimal Transport that is highly correlated with the model's performance on unseen domains.
  arXiv  Detail & Related papers  (2024-05-02T16:35:07Z)
• SPADES: A Realistic Spacecraft Pose Estimation Dataset using Event Sensing [9.583223655096077]
  Due to limited access to real target datasets, algorithms are often trained using synthetic data and applied in the real domain. Event sensing has been explored in the past and shown to reduce the domain gap between simulations and real-world scenarios. We introduce a novel dataset, SPADES, comprising real event data acquired in a controlled laboratory environment and simulated event data using the same camera intrinsics.
  arXiv  Detail & Related papers  (2023-11-09T12:14:47Z)
• Exploring Few-Shot Adaptation for Activity Recognition on Diverse Domains [46.26074225989355]
  Domain adaptation is essential for activity recognition to ensure accurate and robust performance across diverse environments. In this work, we focus on FewShot Domain Adaptation for Activity Recognition (FSDA-AR), which leverages a very small amount of labeled target videos. We propose a new FSDA-AR using five established datasets considering the adaptation on more diverse and challenging domains.
  arXiv  Detail & Related papers  (2023-05-15T08:01:05Z)
• One-Shot Domain Adaptive and Generalizable Semantic Segmentation with Class-Aware Cross-Domain Transformers [96.51828911883456]
  Unsupervised sim-to-real domain adaptation (UDA) for semantic segmentation aims to improve the real-world test performance of a model trained on simulated data. Traditional UDA often assumes that there are abundant unlabeled real-world data samples available during training for the adaptation. We explore the one-shot unsupervised sim-to-real domain adaptation (OSUDA) and generalization problem, where only one real-world data sample is available.
  arXiv  Detail & Related papers  (2022-12-14T15:54:15Z)
• Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection [85.11649974840758]
  3D object detection networks tend to be biased towards the data they are trained on. We propose a single-frame approach for source-free, unsupervised domain adaptation of lidar-based 3D object detectors.
  arXiv  Detail & Related papers  (2021-11-30T18:42:42Z)
• VisDA-2021 Competition Universal Domain Adaptation to Improve Performance on Out-of-Distribution Data [64.91713686654805]
  The Visual Domain Adaptation (VisDA) 2021 competition tests models' ability to adapt to novel test distributions. We will evaluate adaptation to novel viewpoints, backgrounds, modalities and degradation in quality. Performance will be measured using a rigorous protocol, comparing to state-of-the-art domain adaptation methods.
  arXiv  Detail & Related papers  (2021-07-23T03:21:51Z)
• Invariant Feature Learning for Sensor-based Human Activity Recognition [11.334750079923428]
  We present an invariant feature learning framework (IFLF) that extracts common information shared across subjects and devices. Experiments demonstrated that IFLF is effective in handling both subject and device diversion across popular open datasets and an in-house dataset.
  arXiv  Detail & Related papers  (2020-12-14T21:56:17Z)
• SensiX: A Platform for Collaborative Machine Learning on the Edge [69.1412199244903]
  We present SensiX, a personal edge platform that stays between sensor data and sensing models. We demonstrate its efficacy in developing motion and audio-based multi-device sensing systems. Our evaluation shows that SensiX offers a 7-13% increase in overall accuracy and up to 30% increase across different environment dynamics at the expense of 3mW power overhead.
  arXiv  Detail & Related papers  (2020-12-04T23:06:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.