Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions

• URL: http://arxiv.org/abs/2208.01100v1
• Date: Mon, 1 Aug 2022 18:59:05 GMT
• Title: Dyadic Movement Synchrony Estimation Under Privacy-preserving Conditions
• Authors: Jicheng Li, Anjana Bhat, Roghayeh Barmaki
• Abstract summary: This paper proposes an ensemble method for movement synchrony estimation under privacy-preserving conditions. Our method relies entirely on publicly shareable, identity-agnostic secondary data, such as skeleton data and optical flow. We validate our method on two datasets: (1) PT13 dataset collected from autism therapy interventions and (2) TASD-2 dataset collected from synchronized diving competitions.
• Score: 7.053333608725945
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Movement synchrony refers to the dynamic temporal connection between the motions of interacting people. The applications of movement synchrony are wide and broad. For example, as a measure of coordination between teammates, synchrony scores are often reported in sports. The autism community also identifies movement synchrony as a key indicator of children's social and developmental achievements. In general, raw video recordings are often used for movement synchrony estimation, with the drawback that they may reveal people's identities. Furthermore, such privacy concern also hinders data sharing, one major roadblock to a fair comparison between different approaches in autism research. To address the issue, this paper proposes an ensemble method for movement synchrony estimation, one of the first deep-learning-based methods for automatic movement synchrony assessment under privacy-preserving conditions. Our method relies entirely on publicly shareable, identity-agnostic secondary data, such as skeleton data and optical flow. We validate our method on two datasets: (1) PT13 dataset collected from autism therapy interventions and (2) TASD-2 dataset collected from synchronized diving competitions. In this context, our method outperforms its counterpart approaches, both deep neural networks and alternatives.

Related papers

• Loose Social-Interaction Recognition in Real-world Therapy Scenarios [10.088521986304976]
  We propose a novel dual-path architecture to capture the loose interaction between two individuals. Our model learns global abstract features from each stream via a CNNs backbone and fuses them using a new Global-Layer-Attention module. We evaluate our model on real-world autism diagnoses such as our Loose-Interaction dataset, and the publicly available Autism dataset for loose interactions.
  arXiv  Detail & Related papers  (2024-09-30T13:11:14Z)
• Quantum synchronization in one-dimensional topological systems [4.680781132156346]
  Synchronization in quantum systems can be induced through applying dissipation on the central sites. Two types of synchronization, stemming from the topological edge states, are characterized by the off-diagonal or diagonal correlations between the boundary sites. Remarkably, the synchronous oscillations maintain steady amplitude and frequency in the thermodynamic limit.
  arXiv  Detail & Related papers  (2024-09-19T09:03:09Z)
• Synchformer: Efficient Synchronization from Sparse Cues [100.89656994681934]
  Our contributions include a novel audio-visual synchronization model, and training that decouples extraction from synchronization modelling. This approach achieves state-of-the-art performance in both dense and sparse settings. We also extend synchronization model training to AudioSet a million-scale 'in-the-wild' dataset, investigate evidence attribution techniques for interpretability, and explore a new capability for synchronization models: audio-visual synchronizability.
  arXiv  Detail & Related papers  (2024-01-29T18:59:55Z)
• ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions [66.87211993793807]
  We present ReMoS, a denoising diffusion based model that synthesizes full body motion of a person in two person interaction scenario. We demonstrate ReMoS across challenging two person scenarios such as pair dancing, Ninjutsu, kickboxing, and acrobatics. We also contribute the ReMoCap dataset for two person interactions containing full body and finger motions.
  arXiv  Detail & Related papers  (2023-11-28T18:59:52Z)
• GestSync: Determining who is speaking without a talking head [67.75387744442727]
  We introduce Gesture-Sync: determining if a person's gestures are correlated with their speech or not. In comparison to Lip-Sync, Gesture-Sync is far more challenging as there is a far looser relationship between the voice and body movement. We show that the model can be trained using self-supervised learning alone, and evaluate its performance on the LRS3 dataset.
  arXiv  Detail & Related papers  (2023-10-08T22:48:30Z)
• On the Audio-visual Synchronization for Lip-to-Speech Synthesis [22.407313748927393]
  We show that the commonly used audio-visual datasets, such as GRID, TCD-TIMIT, and Lip2Wav, can have data asynchrony issues. Training lip-to-speech with such datasets may further cause the model asynchrony issue -- that is, the generated speech and the input video are out of sync.
  arXiv  Detail & Related papers  (2023-03-01T13:35:35Z)
• Pose Uncertainty Aware Movement Synchrony Estimation via Spatial-Temporal Graph Transformer [7.053333608725945]
  Movement synchrony reflects the coordination of body movements between interacting dyads. This paper proposes a skeleton-based graph transformer for movement synchrony estimation. Our method achieved an overall accuracy of 88.98% and surpassed its counterparts by a wide margin.
  arXiv  Detail & Related papers  (2022-08-01T22:35:32Z)
• Edge Continual Learning for Dynamic Digital Twins over Wireless Networks [68.65520952712914]
  Digital twins (DTs) constitute a critical link between the real-world and the metaverse. In this paper, a novel edge continual learning framework is proposed to accurately model the evolving affinity between a physical twin and its corresponding cyber twin. The proposed framework achieves a simultaneously accurate and synchronous CT model that is robust to catastrophic forgetting.
  arXiv  Detail & Related papers  (2022-04-10T23:25:37Z)
• Learning Iterative Robust Transformation Synchronization [71.73273007900717]
  We propose to use graph neural networks (GNNs) to learn transformation synchronization. In this work, we avoid handcrafting robust loss functions, and propose to use graph neural networks (GNNs) to learn transformation synchronization.
  arXiv  Detail & Related papers  (2021-11-01T07:03:14Z)
• Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition [111.87412719773889]
  We propose a joint learning framework for "interacted object localization" and "human action recognition" based on skeleton data. Our method achieves the best or competitive performance with the state-of-the-art methods for human action recognition.
  arXiv  Detail & Related papers  (2021-10-28T10:09:34Z)
• Automatic audiovisual synchronisation for ultrasound tongue imaging [35.60751372748571]
  Ultrasound and speech audio are recorded simultaneously, and in order to correctly use this data, the two modalities should be correctly synchronised. Synchronisation is achieved using specialised hardware at recording time, but this approach can fail in practice resulting in data of limited usability. In this paper, we address the problem of automatically synchronising ultrasound and audio after data collection. We describe our approach for automatic synchronisation, which is driven by a self-supervised neural network, exploiting the correlation between the two signals to synchronise them.
  arXiv  Detail & Related papers  (2021-05-31T17:11:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.