Pose-Based Sign Language Appearance Transfer

• URL: http://arxiv.org/abs/2410.13675v1
• Date: Thu, 17 Oct 2024 15:33:54 GMT
• Title: Pose-Based Sign Language Appearance Transfer
• Authors: Amit Moryossef, Gerard Sant, Zifan Jiang,
• Abstract summary: We introduce a method for transferring the signer's appearance in sign language skeletal poses while preserving the sign content. This approach improves pose-based rendering and sign stitching while obfuscating identity. Our experiments show that while the method reduces signer identification accuracy, it slightly harms sign recognition performance.
• Score: 5.839722619084469
• License:
• Abstract: We introduce a method for transferring the signer's appearance in sign language skeletal poses while preserving the sign content. Using estimated poses, we transfer the appearance of one signer to another, maintaining natural movements and transitions. This approach improves pose-based rendering and sign stitching while obfuscating identity. Our experiments show that while the method reduces signer identification accuracy, it slightly harms sign recognition performance, highlighting a tradeoff between privacy and utility. Our code is available at \url{https://github.com/sign-language-processing/pose-anonymization}.

Related papers

• Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition [96.62264528407863]
  We propose a self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency. Inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling. Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin.
  arXiv  Detail & Related papers  (2024-06-15T04:50:19Z)
• SignMusketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale [22.49602248323602]
  A persistent challenge in sign language video processing is how we learn representations of sign language. Our proposed method focuses on just the most relevant parts in a signing video: the face, hands and body posture of the signer. Our approach is based on learning from individual frames (rather than video sequences) and is therefore much more efficient than prior work on sign language pre-training.
  arXiv  Detail & Related papers  (2024-06-11T03:00:41Z)
• Sign Stitching: A Novel Approach to Sign Language Production [35.35777909051466]
  We propose using dictionary examples to create expressive sign language sequences. We present a 7-step approach to effectively stitch the signs together. We leverage the SignGAN model to map the output to a photo-realistic signer.
  arXiv  Detail & Related papers  (2024-05-13T11:44:57Z)
• Learnt Contrastive Concept Embeddings for Sign Recognition [33.72708697077754]
  We focus on explicitly creating sign embeddings that bridge the gap between sign language and spoken language. We train a vocabulary of embeddings that are based on the linguistic labels for sign video. We develop a conceptual similarity loss which is able to utilise word embeddings from NLP methods to create sign embeddings that have better sign language to spoken language correspondence.
  arXiv  Detail & Related papers  (2023-08-18T12:47:18Z)
• PerSign: Personalized Bangladeshi Sign Letters Synthesis [0.0]
  Bangladeshi Sign Language (BdSL) is tough to learn for general people. We propose PerSign, a system that can reproduce a person's image by introducing sign gestures in it.
  arXiv  Detail & Related papers  (2022-09-29T07:07:34Z)
• Automatic dense annotation of large-vocabulary sign language videos [85.61513254261523]
  We propose a simple, scalable framework to vastly increase the density of automatic annotations. We make these annotations publicly available to support the sign language research community.
  arXiv  Detail & Related papers  (2022-08-04T17:55:09Z)
• AnonySIGN: Novel Human Appearance Synthesis for Sign Language Video Anonymisation [37.679114155300084]
  We introduce the task of Sign Language Video Anonymisation (SLVA) as an automatic method to anonymise the visual appearance of a sign language video. To tackle SLVA, we propose AnonySign, a novel automatic approach for visual anonymisation of sign language data.
  arXiv  Detail & Related papers  (2021-07-22T13:42:18Z)
• Read and Attend: Temporal Localisation in Sign Language Videos [84.30262812057994]
  We train a Transformer model to ingest a continuous signing stream and output a sequence of written tokens. We show that it acquires the ability to attend to a large vocabulary of sign instances in the input sequence, enabling their localisation.
  arXiv  Detail & Related papers  (2021-03-30T16:39:53Z)
• Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
  Sign language is used by deaf or speech impaired people to communicate. Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance. Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
  arXiv  Detail & Related papers  (2021-03-16T03:38:17Z)
• BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues [106.21067543021887]
  We show how to use mouthing cues from signers to obtain high-quality annotations from video data. The BSL-1K dataset is a collection of British Sign Language (BSL) signs of unprecedented scale.
  arXiv  Detail & Related papers  (2020-07-23T16:59:01Z)
• Transferring Cross-domain Knowledge for Video Sign Language Recognition [103.9216648495958]
  Word-level sign language recognition (WSLR) is a fundamental task in sign language interpretation. We propose a novel method that learns domain-invariant visual concepts and fertilizes WSLR models by transferring knowledge of subtitled news sign to them.
  arXiv  Detail & Related papers  (2020-03-08T03:05:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.