Shared Representation for 3D Pose Estimation, Action Classification, and Progress Prediction from Tactile Signals

• URL: http://arxiv.org/abs/2603.25906v1
• Date: Thu, 26 Mar 2026 20:55:45 GMT
• Title: Shared Representation for 3D Pose Estimation, Action Classification, and Progress Prediction from Tactile Signals
• Abstract summary: Estimating human pose, classifying actions, and predicting movement progress are essential for human-robot interaction.<n>Prior tactile-based approaches handle each task separately, leading to suboptimal performance.<n>We propose a Shared COnvolutional Transformer for Tactile Inference (SCOTTI) that learns a shared representation to simultaneously address three separate prediction tasks.
• Score: 11.658733717166205
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating human pose, classifying actions, and predicting movement progress are essential for human-robot interaction. While vision-based methods suffer from occlusion and privacy concerns in realistic environments, tactile sensing avoids these issues. However, prior tactile-based approaches handle each task separately, leading to suboptimal performance. In this study, we propose a Shared COnvolutional Transformer for Tactile Inference (SCOTTI) that learns a shared representation to simultaneously address three separate prediction tasks: 3D human pose estimation, action class categorization, and action completion progress estimation. To the best of our knowledge, this is the first work to explore action progress prediction using foot tactile signals from custom wireless insole sensors. This unified approach leverages the mutual benefits of multi-task learning, enabling the model to achieve improved performance across all three tasks compared to learning them independently. Experimental results demonstrate that SCOTTI outperforms existing approaches across all three tasks. Additionally, we introduce a novel dataset collected from 15 participants performing various activities and exercises, with 7 hours of total duration, across eight different activities.

Related papers

• Object Affordance Recognition and Grounding via Multi-scale Cross-modal Representation Learning [64.32618490065117]
  A core problem of Embodied AI is to learn object manipulation from observation, as humans do.<n>We propose a novel approach that learns an affordance-aware 3D representation and employs a stage-wise inference strategy.<n> Experiments demonstrate the effectiveness of our method, showing improved performance in both affordance grounding and classification.
  arXiv  Detail & Related papers  (2025-08-02T04:14:18Z)
• Learning Manipulation by Predicting Interaction [85.57297574510507]
  We propose a general pre-training pipeline that learns Manipulation by Predicting the Interaction. The experimental results demonstrate that MPI exhibits remarkable improvement by 10% to 64% compared with previous state-of-the-art in real-world robot platforms.
  arXiv  Detail & Related papers  (2024-06-01T13:28:31Z)
• Self-supervised New Activity Detection in Sensor-based Smart Environments [2.5486448837945765]
  We introduce CLAN, a model that leverages Contrastive Learning with diverse data Augmentation for New activity detection.<n>CLAN incorporates both time and frequency domains, enabling the learning of multi-faceted discriminative representations.<n>CLAN achieves a 9.24% improvement in AUROC compared to the best-performing baseline model.
  arXiv  Detail & Related papers  (2024-01-17T03:57:36Z)
• Disentangled Interaction Representation for One-Stage Human-Object Interaction Detection [70.96299509159981]
  Human-Object Interaction (HOI) detection is a core task for human-centric image understanding. Recent one-stage methods adopt a transformer decoder to collect image-wide cues that are useful for interaction prediction. Traditional two-stage methods benefit significantly from their ability to compose interaction features in a disentangled and explainable manner.
  arXiv  Detail & Related papers  (2023-12-04T08:02:59Z)
• What Makes Pre-Trained Visual Representations Successful for Robust Manipulation? [57.92924256181857]
  We find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models.
  arXiv  Detail & Related papers  (2023-11-03T18:09:08Z)
• SPOT: Scalable 3D Pre-training via Occupancy Prediction for Learning Transferable 3D Representations [76.45009891152178]
  Pretraining-finetuning approach can alleviate the labeling burden by fine-tuning a pre-trained backbone across various downstream datasets as well as tasks.<n>We show, for the first time, that general representations learning can be achieved through the task of occupancy prediction.<n>Our findings will facilitate the understanding of LiDAR points and pave the way for future advancements in LiDAR pre-training.
  arXiv  Detail & Related papers  (2023-09-19T11:13:01Z)
• Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints [25.550524178542833]
  We propose a novel multi-task learning framework for pedestrian crossing action recognition and trajectory prediction. We use 3D human keypoints extracted from raw sensor data to capture rich information on human pose and activity. We show that our approach achieves state-of-the-art performance on a wide range of evaluation metrics.
  arXiv  Detail & Related papers  (2023-06-01T18:27:48Z)
• Self-Supervised Human Activity Recognition with Localized Time-Frequency Contrastive Representation Learning [16.457778420360537]
  We propose a self-supervised learning solution for human activity recognition with smartphone accelerometer data. We develop a model that learns strong representations from accelerometer signals, while reducing the model's reliance on class labels. We evaluate the performance of the proposed solution on three datasets, namely MotionSense, HAPT, and HHAR.
  arXiv  Detail & Related papers  (2022-08-26T22:47:18Z)
• Improving Human Motion Prediction Through Continual Learning [2.720960618356385]
  Human motion prediction is an essential component for enabling closer human-robot collaboration. It is compounded by the variability of human motion, both at a skeletal level due to the varying size of humans and at a motion level due to individual movement idiosyncrasies. We propose a modular sequence learning approach that allows end-to-end training while also having the flexibility of being fine-tuned.
  arXiv  Detail & Related papers  (2021-07-01T15:34:41Z)
• Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation [84.28064034301445]
  Self-similarity, and the resulting ambiguities in assigning pixel observations to the respective hands, is a major cause of the final 3D pose error. We propose DIGIT, a novel method for estimating the 3D poses of two interacting hands from a single monocular image. We experimentally show that the proposed approach achieves new state-of-the-art performance on the InterHand2.6M dataset.
  arXiv  Detail & Related papers  (2021-07-01T13:28:02Z)
• Pedestrian Behavior Prediction via Multitask Learning and Categorical Interaction Modeling [13.936894582450734]
  We propose a multitask learning framework that simultaneously predicts trajectories and actions of pedestrians by relying on multimodal data. We show that our model achieves state-of-the-art performance and improves trajectory and action prediction by up to 22% and 6% respectively.
  arXiv  Detail & Related papers  (2020-12-06T15:57:11Z)
• Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion [16.457778420360537]
  We propose the use of self-supervised learning for human activity recognition with smartphone accelerometer data. First, the representations of unlabeled input signals are learned by training a deep convolutional neural network to predict a segment of accelerometer values. For this task, we add a number of fully connected layers to the end of the frozen network and train the added layers with labeled accelerometer signals to learn to classify human activities.
  arXiv  Detail & Related papers  (2020-10-21T02:14:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.