Diversity-Aware Sign Language Production through a Pose Encoding Variational Autoencoder

• URL: http://arxiv.org/abs/2405.10423v1
• Date: Thu, 16 May 2024 20:04:35 GMT
• Title: Diversity-Aware Sign Language Production through a Pose Encoding Variational Autoencoder
• Abstract summary: We extend the variational inference paradigm to include information about the pose and the conditioning of the attributes. The generator framework is presented as a UNet architecture to ensure spatial preservation of the input pose. Experiments on the SMILE II dataset show that the proposed model performs quantitatively better than state-of-the-art baselines.
• Score: 33.20529988919208
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper addresses the problem of diversity-aware sign language production, where we want to give an image (or sequence) of a signer and produce another image with the same pose but different attributes (\textit{e.g.} gender, skin color). To this end, we extend the variational inference paradigm to include information about the pose and the conditioning of the attributes. This formulation improves the quality of the synthesised images. The generator framework is presented as a UNet architecture to ensure spatial preservation of the input pose, and we include the visual features from the variational inference to maintain control over appearance and style. We generate each body part with a separate decoder. This architecture allows the generator to deliver better overall results. Experiments on the SMILE II dataset show that the proposed model performs quantitatively better than state-of-the-art baselines regarding diversity, per-pixel image quality, and pose estimation. Quantitatively, it faithfully reproduces non-manual features for signers.

Related papers

• Towards Generalized Multi-Image Editing for Unified Multimodal Models [56.620038824933566]
  Unified Multimodal Models (UMMs) integrate multimodal understanding and generation.<n>UMMs are limited to maintaining visual consistency and disambiguating visual cues when referencing details across multiple input images.<n>We propose a scalable multi-image editing framework for UMMs that explicitly distinguishes image identities and generalizes to variable input counts.
  arXiv  Detail & Related papers  (2026-01-09T06:42:49Z)
• Omni-Attribute: Open-vocabulary Attribute Encoder for Visual Concept Personalization [82.31106470150844]
  We introduce Omni-Attribute, the first open-vocabulary image attribute encoder to learn attribute-specific representations.<n>We use a dual-objective training paradigm that balances generative fidelity with contrastive disentanglement.<n>The resulting embeddings prove effective for open-vocabulary attribute retrieval, personalization, and compositional generation.
  arXiv  Detail & Related papers  (2025-12-11T18:59:56Z)
• Diverse Signer Avatars with Manual and Non-Manual Feature Modelling for Sign Language Production [20.773821519253534]
  We show that our proposed module ensures the preservation of linguistic content while seamlessly using reference images with different ethnic backgrounds to ensure diversity.<n> Experiments on the YouTube-SL-25 sign language dataset show that our pipeline achieves superior visual quality compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-08-21T22:16:32Z)
• Object-level Visual Prompts for Compositional Image Generation [75.6085388740087]
  We introduce a method for composing object-level visual prompts within a text-to-image diffusion model. A key challenge in this task is to preserve the identity of the objects depicted in the input visual prompts. We introduce a new KV-mixed cross-attention mechanism, in which keys and values are learned from distinct visual representations.
  arXiv  Detail & Related papers  (2025-01-02T18:59:44Z)
• ImageFolder: Autoregressive Image Generation with Folded Tokens [51.815319504939396]
  Increasing token length is a common approach to improve the image reconstruction quality. There exists a trade-off between reconstruction and generation quality regarding token length. We propose Image, a semantic tokenizer that provides spatially aligned image tokens that can be folded during autoregressive modeling.
  arXiv  Detail & Related papers  (2024-10-02T17:06:39Z)
• Imagine yourself: Tuning-Free Personalized Image Generation [39.63411174712078]
  We introduce Imagine yourself, a state-of-the-art model designed for personalized image generation. It operates as a tuning-free model, enabling all users to leverage a shared framework without individualized adjustments. Our study demonstrates that Imagine yourself surpasses the state-of-the-art personalization model, exhibiting superior capabilities in identity preservation, visual quality, and text alignment.
  arXiv  Detail & Related papers  (2024-09-20T09:21:49Z)
• EZIGen: Enhancing zero-shot personalized image generation with precise subject encoding and decoupled guidance [20.430259028981094]
  EZIGen aims to produce images that align with both a given text prompt and subject image. It employs two main components: a carefully crafted subject image encoder based on the pre-trained UNet of the Stable Diffusion model. It achieves state-of-the-art results on multiple personalized generation benchmarks with a unified model and 100 times less training data.
  arXiv  Detail & Related papers  (2024-09-12T14:44:45Z)
• Img-Diff: Contrastive Data Synthesis for Multimodal Large Language Models [49.439311430360284]
  We introduce a novel data synthesis method inspired by contrastive learning and image difference captioning. Our key idea involves challenging the model to discern both matching and distinct elements. We leverage this generated dataset to fine-tune state-of-the-art (SOTA) MLLMs.
  arXiv  Detail & Related papers  (2024-08-08T17:10:16Z)
• Language Guided Domain Generalized Medical Image Segmentation [68.93124785575739]
  Single source domain generalization holds promise for more reliable and consistent image segmentation across real-world clinical settings. We propose an approach that explicitly leverages textual information by incorporating a contrastive learning mechanism guided by the text encoder features. Our approach achieves favorable performance against existing methods in literature.
  arXiv  Detail & Related papers  (2024-04-01T17:48:15Z)
• Cross-Image Attention for Zero-Shot Appearance Transfer [68.43651329067393]
  We introduce a cross-image attention mechanism that implicitly establishes semantic correspondences across images. We harness three mechanisms that either manipulate the noisy latent codes or the model's internal representations throughout the denoising process. Experiments show that our method is effective across a wide range of object categories and is robust to variations in shape, size, and viewpoint.
  arXiv  Detail & Related papers  (2023-11-06T18:33:24Z)
• On the Suitability of Representations for Quality Diversity Optimization of Shapes [77.34726150561087]
  The representation, or encoding, utilized in evolutionary algorithms has a substantial effect on their performance. This study compares the impact of several representations, including direct encoding, a dictionary-based representation, parametric encoding, compositional pattern producing networks, and cellular automata, on the generation of voxelized meshes.
  arXiv  Detail & Related papers  (2023-04-07T07:34:23Z)
• Improving Visual Quality of Image Synthesis by A Token-based Generator with Transformers [51.581926074686535]
  We present a new perspective of achieving image synthesis by viewing this task as a visual token generation problem. The proposed TokenGAN has achieved state-of-the-art results on several widely-used image synthesis benchmarks.
  arXiv  Detail & Related papers  (2021-11-05T12:57:50Z)
• Improving Generation and Evaluation of Visual Stories via Semantic Consistency [72.00815192668193]
  Given a series of natural language captions, an agent must generate a sequence of images that correspond to the captions. Prior work has introduced recurrent generative models which outperform synthesis text-to-image models on this task. We present a number of improvements to prior modeling approaches, including the addition of a dual learning framework.
  arXiv  Detail & Related papers  (2021-05-20T20:42:42Z)
• Generating Annotated High-Fidelity Images Containing Multiple Coherent Objects [10.783993190686132]
  We propose a multi-object generation framework that can synthesize images with multiple objects without explicitly requiring contextual information. We demonstrate how coherency and fidelity are preserved with our method through experiments on the Multi-MNIST and CLEVR datasets.
  arXiv  Detail & Related papers  (2020-06-22T11:33:55Z)
• Controllable Person Image Synthesis with Attribute-Decomposed GAN [27.313729413684012]
  This paper introduces the Attribute-Decomposed GAN, a novel generative model for controllable person image synthesis. The core idea of the proposed model is to embed human attributes into the latent space as independent codes. Experimental results demonstrate the proposed method's superiority over the state of the art in pose transfer.
  arXiv  Detail & Related papers  (2020-03-27T07:47:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.