TapNet: The Design, Training, Implementation, and Applications of a Multi-Task Learning CNN for Off-Screen Mobile Input

• URL: http://arxiv.org/abs/2102.09087v1
• Date: Thu, 18 Feb 2021 00:45:41 GMT
• Title: TapNet: The Design, Training, Implementation, and Applications of a Multi-Task Learning CNN for Off-Screen Mobile Input
• Authors: Michael Xuelin Huang, Yang Li, Nazneen Nazneen, Alexander Chao, Shumin Zhai
• Abstract summary: We present the design, training, implementation and applications of TapNet, a multi-task network that detects tapping on the smartphone. TapNet can jointly learn from data across devices and simultaneously recognize multiple tap properties, including tap direction and tap location.
• Score: 75.05709030478073
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To make off-screen interaction without specialized hardware practical, we investigate using deep learning methods to process the common built-in IMU sensor (accelerometers and gyroscopes) on mobile phones into a useful set of one-handed interaction events. We present the design, training, implementation and applications of TapNet, a multi-task network that detects tapping on the smartphone. With phone form factor as auxiliary information, TapNet can jointly learn from data across devices and simultaneously recognize multiple tap properties, including tap direction and tap location. We developed two datasets consisting of over 135K training samples, 38K testing samples, and 32 participants in total. Experimental evaluation demonstrated the effectiveness of the TapNet design and its significant improvement over the state of the art. Along with the datasets, (https://sites.google.com/site/michaelxlhuang/datasets/tapnet-dataset), and extensive experiments, TapNet establishes a new technical foundation for off-screen mobile input.

Related papers

• MuJo: Multimodal Joint Feature Space Learning for Human Activity Recognition [2.7532797256542403]
  Human Activity Recognition (HAR) is a longstanding problem in AI with applications in a broad range of areas, including healthcare, sports and fitness, security, and more. We introduce our comprehensive Fitness Multimodal Activity dataset (FiMAD) to enhance HAR performance across various modalities. We show that classifiers pre-trained on FiMAD can increase the performance on real HAR datasets such as MM-Fit, MyoGym, MotionSense, and MHEALTH.
  arXiv  Detail & Related papers  (2024-06-06T08:42:36Z)
• Learning Visuotactile Skills with Two Multifingered Hands [80.99370364907278]
  We explore learning from human demonstrations using a bimanual system with multifingered hands and visuotactile data. Our results mark a promising step forward in bimanual multifingered manipulation from visuotactile data.
  arXiv  Detail & Related papers  (2024-04-25T17:59:41Z)
• FedOpenHAR: Federated Multi-Task Transfer Learning for Sensor-Based Human Activity Recognition [0.0]
  This paper explores Federated Transfer Learning in a Multi-Task manner for both sensor-based human activity recognition and device position identification tasks. The OpenHAR framework is used to train the models, which contains ten smaller datasets. By utilizing transfer learning and training a task-specific and personalized federated model, we obtained a similar accuracy with training each client individually and higher accuracy than a fully centralized approach.
  arXiv  Detail & Related papers  (2023-11-13T21:31:07Z)
• Your Identity is Your Behavior -- Continuous User Authentication based on Machine Learning and Touch Dynamics [0.0]
  This research used a dataset of touch dynamics collected from 40 subjects using the LG V30+. The participants played four mobile games, Diep.io, Slither, and Minecraft, for 10 minutes each game. The results of the research showed that all three algorithms were able to effectively classify users based on their individual touch dynamics.
  arXiv  Detail & Related papers  (2023-04-24T13:45:25Z)
• Simultaneous prediction of hand gestures, handedness, and hand keypoints using thermal images [0.6087960723103347]
  We propose a technique for simultaneous hand gesture classification, handedness detection, and hand keypoints localization using thermal data captured by an infrared camera. Our method uses a novel deep multi-task learning architecture that includes shared encoderdecoder layers followed by three branches dedicated for each mentioned task.
  arXiv  Detail & Related papers  (2023-03-02T19:25:40Z)
• Online Data Selection for Federated Learning with Limited Storage [53.46789303416799]
  Federated Learning (FL) has been proposed to achieve distributed machine learning among networked devices. The impact of on-device storage on the performance of FL is still not explored. In this work, we take the first step to consider the online data selection for FL with limited on-device storage.
  arXiv  Detail & Related papers  (2022-09-01T03:27:33Z)
• A Wireless-Vision Dataset for Privacy Preserving Human Activity Recognition [53.41825941088989]
  A new WiFi-based and video-based neural network (WiNN) is proposed to improve the robustness of activity recognition. Our results show that WiVi data set satisfies the primary demand and all three branches in the proposed pipeline keep more than $80%$ of activity recognition accuracy.
  arXiv  Detail & Related papers  (2022-05-24T10:49:11Z)
• Mobile Behavioral Biometrics for Passive Authentication [65.94403066225384]
  This work carries out a comparative analysis of unimodal and multimodal behavioral biometric traits. Experiments are performed over HuMIdb, one of the largest and most comprehensive freely available mobile user interaction databases. In our experiments, the most discriminative background sensor is the magnetometer, whereas among touch tasks the best results are achieved with keystroke.
  arXiv  Detail & Related papers  (2022-03-14T17:05:59Z)
• Exploring System Performance of Continual Learning for Mobile and Embedded Sensing Applications [19.334890205028568]
  We conduct the first comprehensive empirical study that quantifies the performance of three predominant continual learning schemes. We implement an end-to-end continual learning framework on edge devices. We demonstrate for the first time that it is feasible and practical to run continual learning on-device with a limited memory budget.
  arXiv  Detail & Related papers  (2021-10-25T22:06:26Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.