Pose-Guided Sign Language Video GAN with Dynamic Lambda

• URL: http://arxiv.org/abs/2105.02742v1
• Date: Thu, 6 May 2021 15:12:09 GMT
• Title: Pose-Guided Sign Language Video GAN with Dynamic Lambda
• Abstract summary: We propose a novel approach for the of sign language videos using GANs. We use the human semantic synthesis of the Soft-Gated Warping-GAN from to produce photorealistic videos guided by region-level spatial layouts.
• Score: 0.41998444721319206
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We propose a novel approach for the synthesis of sign language videos using GANs. We extend the previous work of Stoll et al. by using the human semantic parser of the Soft-Gated Warping-GAN from to produce photorealistic videos guided by region-level spatial layouts. Synthesizing target poses improves performance on independent and contrasting signers. Therefore, we have evaluated our system with the highly heterogeneous MS-ASL dataset with over 200 signers resulting in a SSIM of 0.893. Furthermore, we introduce a periodic weighting approach to the generator that reactivates the training and leads to quantitatively better results.

Related papers

• Probing Geospatial SSL Representations with Environmental Signals [51.908606610432194]
  Self-supervised learning (SSL) is designed to learn generic, transferable representations rather than representations optimized for a single task.<n>We ask a different question: do SSL representations preserve statistical associations with environmental variables that co-vary with the imaging process?<n>To answer this question, we probe SSL representations using co-located ERA5 reanalysis variables, a global dataset of physically consistent environmental variables.
  arXiv  Detail & Related papers  (2026-07-06T15:24:25Z)
• XLSR-Kanformer: A KAN-Intergrated model for Synthetic Speech Detection [1.8334165690218873]
  We propose a novel approach that replaces the traditional Multi-Layer Perceptron (MLP) in the XLSR-Conformer model with a Kolmogorov-Arnold Network (KAN)<n>Our experimental results demonstrate that the integration of KAN to XLSR-Conformer model can improve the performance by 60.55% relatively in Equal Error Rate (EER) LA and DF sets.
  arXiv  Detail & Related papers  (2025-10-08T06:58:58Z)
• Pushing the Performance of Synthetic Speech Detection with Kolmogorov-Arnold Networks and Self-Supervised Learning Models [1.7205106391379026]
  We propose a novel approach that replaces the traditional Multi-Layer Perceptron in the XLSR-Conformer model with a Kolmogorov-Arnold Network (KAN)<n>Our results show that integrating KAN into the SSL-based models can improve the performance by 60.55% relatively on LA and DF sets.
  arXiv  Detail & Related papers  (2025-06-17T03:30:58Z)
• Learning Beyond Limits: Multitask Learning and Synthetic Data for Low-Resource Canonical Morpheme Segmentation [7.766518675734386]
  We introduce a transformer-based morpheme segmentation system that augments a low-resource training signal.<n>Our framework jointly predicts morphological segments and glosses from orthographic input.<n>We integrate synthetic training data generated by large language models (LLMs) using in-context learning.
  arXiv  Detail & Related papers  (2025-05-22T15:40:09Z)
• Beyond Sample-Level Feedback: Using Reference-Level Feedback to Guide Data Synthesis [55.65459867300319]
  LLMs demonstrate remarkable capabilities in following natural language instructions, largely due to instruction-tuning on high-quality datasets.<n>Recent approaches incorporate feedback to improve data quality, but typically operate at the sample level, generating and applying feedback for each response individually.<n>We propose Reference-Level Feedback, a novel methodology that instead collects feedback based on high-quality reference samples from carefully curated seed data.
  arXiv  Detail & Related papers  (2025-02-06T21:29:00Z)
• LS-GAN: Human Motion Synthesis with Latent-space GANs [0.0]
  Conditioned Motion synthesis takes a text input and outputs a 3D motion corresponding to the text. We introduce a novel framework that utilizes Generative Adversarial Networks (GANs) in the latent space to enable faster training and inference.
  arXiv  Detail & Related papers  (2024-12-30T05:44:38Z)
• Training Strategies for Isolated Sign Language Recognition [72.27323884094953]
  This paper introduces a comprehensive model training pipeline for Isolated Sign Language Recognition. The constructed pipeline incorporates carefully selected image and video augmentations to tackle the challenges of low data quality and varying sign speeds. We achieve a state-of-the-art result on the WLASL and Slovo benchmarks with 1.63% and 14.12% improvements compared to the previous best solution.
  arXiv  Detail & Related papers  (2024-12-16T08:37:58Z)
• Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback [50.84142264245052]
  This work introduces the Align-SLM framework to enhance the semantic understanding of textless Spoken Language Models (SLMs) Our approach generates multiple speech continuations from a given prompt and uses semantic metrics to create preference data for Direct Preference Optimization (DPO) We evaluate the framework using ZeroSpeech 2021 benchmarks for lexical and syntactic modeling, the spoken version of the StoryCloze dataset for semantic coherence, and other speech generation metrics, including the GPT4-o score and human evaluation.
  arXiv  Detail & Related papers  (2024-11-04T06:07:53Z)
• Learning from Synthetic Data for Visual Grounding [55.21937116752679]
  We show that SynGround can improve the localization capabilities of off-the-shelf vision-and-language models. Data generated with SynGround improves the pointing game accuracy of a pretrained ALBEF and BLIP models by 4.81% and 17.11% absolute percentage points, respectively.
  arXiv  Detail & Related papers  (2024-03-20T17:59:43Z)
• Aligning Large Language Models through Synthetic Feedback [43.84431341195111]
  We propose a novel alignment learning framework with synthetic feedback not dependent on extensive human annotations. In human evaluation, our model is preferred to Alpaca and Dolly-v2, 55.0% and 58.5% of the time, respectively.
  arXiv  Detail & Related papers  (2023-05-23T06:41:16Z)
• Leaping Into Memories: Space-Time Deep Feature Synthesis [93.10032043225362]
  We propose LEAPS, an architecture-independent method for synthesizing videos from internal models. We quantitatively and qualitatively evaluate the applicability of LEAPS by inverting a range of architectures convolutional attention-based on Kinetics-400.
  arXiv  Detail & Related papers  (2023-03-17T12:55:22Z)
• A Complete Recipe for Diffusion Generative Models [18.891215475887314]
  We present a complete recipe for formulating forward processes in Generative Models (SGMs) We introduce Phase Space Langevin Diffusion (PSLD), which relies on score-based modeling within an augmented space enriched by auxiliary variables.
  arXiv  Detail & Related papers  (2023-03-03T07:20:58Z)
• Multi-Modal Zero-Shot Sign Language Recognition [51.07720650677784]
  We propose a multi-modal Zero-Shot Sign Language Recognition model. A Transformer-based model along with a C3D model is used for hand detection and deep features extraction. A semantic space is used to map the visual features to the lingual embedding of the class labels.
  arXiv  Detail & Related papers  (2021-09-02T09:10:39Z)
• Score-based Generative Modeling in Latent Space [93.8985523558869]
  Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution coverage. Here, we propose the Latent Score-based Generative Model (LSGM), a novel approach that trains SGMs in a latent space. Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space.
  arXiv  Detail & Related papers  (2021-06-10T17:26:35Z)
• Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognition [9.30315673109153]
  We introduce SynSE, a novel syntactically guided generative approach for Zero-Shot Learning (ZSL) Our end-to-end approach learns progressively refined generative embedding spaces constrained within and across the involved modalities (visual, language) Our results demonstrate SynSE's state of the art performance in both ZSL and Generalized Zero-Shot Learning (GZSL) settings.
  arXiv  Detail & Related papers  (2021-01-27T16:34:27Z)
• Improving Augmentation and Evaluation Schemes for Semantic Image Synthesis [16.097324852253912]
  We introduce a novel augmentation scheme designed specifically for generative adversarial networks (GANs) We propose to randomly warp object shapes in the semantic label maps used as an input to the generator. The local shape discrepancies between the warped and non-warped label maps and images enable the GAN to learn better the structural and geometric details of the scene.
  arXiv  Detail & Related papers  (2020-11-25T10:55:26Z)
• Few-Shot Adaptation of Generative Adversarial Networks [54.014885321880755]
  This paper proposes a simple and effective method, Few-Shot GAN, for adapting GANs in few-shot settings (less than 100 images) FSGAN learns to adapt the singular values of the pre-trained weights while freezing the corresponding singular vectors. We show that our method has significant visual quality gains compared with existing GAN adaptation methods.
  arXiv  Detail & Related papers  (2020-10-22T17:59:29Z)
• Synthetic Convolutional Features for Improved Semantic Segmentation [139.5772851285601]
  We suggest to generate intermediate convolutional features and propose the first synthesis approach that is catered to such intermediate convolutional features. This allows us to generate new features from label masks and include them successfully into the training procedure. Experimental results and analysis on two challenging datasets Cityscapes and ADE20K show that our generated feature improves performance on segmentation tasks.
  arXiv  Detail & Related papers  (2020-09-18T14:12:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.