Human-Centered Editable Speech-to-Sign-Language Generation via Streaming Conformer-Transformer and Resampling Hook

• URL: http://arxiv.org/abs/2506.14677v2
• Date: Tue, 24 Jun 2025 13:11:34 GMT
• Title: Human-Centered Editable Speech-to-Sign-Language Generation via Streaming Conformer-Transformer and Resampling Hook
• Abstract summary: Existing end-to-end sign-language animation systems suffer from low naturalness, limited facial/body expressivity, and no user control.<n>We propose a human-centered, real-time speech-to-sign animation framework.<n> Deployed on Unity3D, our system achieves a 13 ms frame-in time and a 103 ms latency.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing end-to-end sign-language animation systems suffer from low naturalness, limited facial/body expressivity, and no user control. We propose a human-centered, real-time speech-to-sign animation framework that integrates (1) a streaming Conformer encoder with an autoregressive Transformer-MDN decoder for synchronized upper-body and facial motion generation, (2) a transparent, editable JSON intermediate representation empowering deaf users and experts to inspect and modify each sign segment, and (3) a human-in-the-loop optimization loop that refines the model based on user edits and ratings. Deployed on Unity3D, our system achieves a 13 ms average frame-inference time and a 103 ms end-to-end latency on an RTX 4070. Our key contributions include the design of a JSON-centric editing mechanism for fine-grained sign-level personalization and the first application of an MDN-based feedback loop for continuous model adaptation. This combination establishes a generalizable, explainable AI paradigm for user-adaptive, low-latency multimodal systems. In studies with 20 deaf signers and 5 professional interpreters, we observe a +13 point SUS improvement, 6.7 point reduction in cognitive load, and significant gains in naturalness and trust (p $<$ .001) over baselines. This work establishes a scalable, explainable AI paradigm for accessible sign-language technologies.

Related papers

• MUGEN: A Unified Framework for Efficient Motion Understanding and Generation [17.452895169020817]
  We propose a unified motion--language framework that pays neither cost: no codebook, one draw.<n>A single adaptive-length autoencoder compresses any-length motion into a few continuous latent slots, the system's only motion representation.<n> MUGEN leads language-model baselines on FID on HumanML3D while raising retrieval precision above the real-motion reference.
  arXiv  Detail & Related papers  (2026-07-30T01:55:51Z)
• OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder [17.494692126897434]
  We present textbf Omni-Encoder, a unified Transformer backbone designed to co-embed visual and audio signals at a symmetrical 25 fps within a shared latent space.<n> Experiments demonstrate that, compared to the modality-specific baseline Qwen2.5- Omni, Omni-Encoder delivers substantial gains on visual continuous understanding tasks.<n>These results suggest that unified omnivorous encoding offers a promising direction for building omni-modal models.
  arXiv  Detail & Related papers  (2026-05-02T15:55:03Z)
• Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model [52.79436545460808]
  We present daVinci-MagiHuman, an open-source audio-video generative foundation model for human-centric generation.<n>DaVinci-MagiHuman jointly generates synchronized video and audio using a single-stream Transformer.<n>It supports multilingual spoken generation across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French.
  arXiv  Detail & Related papers  (2026-03-23T13:49:06Z)
• SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learning [54.232148007248874]
  Current Sign Language Production (SLP) frameworks face a stark trade-off: direct text-to-pose models suffer from regression-to-the-mean effects.<n>We propose a novel training paradigm that leverages sparses to capture the true underlying distribution of human signing.<n>By predicting dense motion from these discrete anchors, our approach mitigates regression-to-the-mean while ensuring fluid articulation.
  arXiv  Detail & Related papers  (2026-03-11T06:02:36Z)
• AR-Omni: A Unified Autoregressive Model for Any-to-Any Generation [60.02195766025208]
  We present AR- Omni, a unified any-to-any model in the autoregressive paradigm without any expert decoders.<n>AR- Omni supports autoregressive text and image generation, as well as streaming speech generation, all under a single Transformer decoder.<n>We address three practical issues in unified AR modeling: modality imbalance via task-aware loss reweighting, visual fidelity via a lightweight token-level perceptual alignment loss for image tokens, and stability-creativity trade-offs via a finite-state decoding mechanism.
  arXiv  Detail & Related papers  (2026-01-25T09:17:36Z)
• ShowUI-$π$: Flow-based Generative Models as GUI Dexterous Hands [59.222064425122795]
  We develop ShowUI-$$, the first flow-based generative model as GUI dexterous hand.<n>ShowUI-$$ achieves 26.98 with only 450M parameters, underscoring both the difficulty of the task and the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-12-31T16:51:14Z)
• One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Framework [51.50565654314582]
  Large language models can follow users' instructions throughout a dialogue spanning multiple topics.<n>Existing benchmarks are often limited to a fixed number of turns, making them susceptible to saturation and failing to account for the user's interactive experience.<n>We propose a framework for assessing multi-turn instruction-following ability.
  arXiv  Detail & Related papers  (2025-11-05T14:39:59Z)
• Neural-Driven Image Editing [51.11173675034121]
  Traditional image editing relies on manual prompting, making it labor-intensive and inaccessible to individuals with limited motor control or language abilities.<n>We propose LoongX, a hands-free image editing approach driven by neurophysiological signals.<n>LoongX utilizes state-of-the-art diffusion models trained on a comprehensive dataset of 23,928 image editing pairs.
  arXiv  Detail & Related papers  (2025-07-07T18:31:50Z)
• Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling [76.23539797803681]
  Existing methods primarily use a look mechanism, relying on future text to achieve natural streaming speech synthesis.<n>We propose LE, a streaming framework for generating high-quality speech frame-by-frame.<n> Experimental results suggest that the LE outperforms current streaming TTS methods and achieves comparable performance over sentence-level TTS systems.
  arXiv  Detail & Related papers  (2025-05-26T08:25:01Z)
• Breaking the Barriers: Video Vision Transformers for Word-Level Sign Language Recognition [41.94295877935867]
  Sign language is a fundamental means of communication for the deaf and hard-of-hearing (DHH) community.<n> barriers persist due to the limited fluency in sign language among the hearing population.<n>We propose a Video Vision Transformer (ViViT) model for word-level American Sign Language (ASL) recognition.
  arXiv  Detail & Related papers  (2025-04-10T14:27:25Z)
• VarGes: Improving Variation in Co-Speech 3D Gesture Generation via StyleCLIPS [4.996271098355553]
  VarGes is a novel variation-driven framework designed to enhance co-speech gesture generation.<n>Our approach is validated on benchmark datasets, where it outperforms existing methods in terms of gesture diversity and naturalness.
  arXiv  Detail & Related papers  (2025-02-15T08:46:01Z)
• USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions [6.2486440301992605]
  We propose User-VLM 360deg, a holistic framework integrating multimodal user modeling with bias-aware optimization.<n>Our approach features: (1) user-aware tuning that adapts interactions in real time using visual-linguistic signals; (2) bias mitigation via preference optimization; and (3) curated 360deg socio-emotive interaction datasets annotated with demographic, emotion, and relational metadata.
  arXiv  Detail & Related papers  (2025-02-15T02:25:49Z)
• Combo: Co-speech holistic 3D human motion generation and efficient customizable adaptation in harmony [55.26315526382004]
  We propose a novel framework, Combo, for co-speech holistic 3D human motion generation. In particular, we identify that one fundamental challenge as the multiple-input-multiple-output nature of the generative model of interest. Combo is highly effective in generating high-quality motions but also efficient in transferring identity and emotion.
  arXiv  Detail & Related papers  (2024-08-18T07:48:49Z)
• Motion-Agent: A Conversational Framework for Human Motion Generation with LLMs [67.59291068131438]
  Motion-Agent is a conversational framework designed for general human motion generation, editing, and understanding. Motion-Agent employs an open-source pre-trained language model to develop a generative agent, MotionLLM, that bridges the gap between motion and text.
  arXiv  Detail & Related papers  (2024-05-27T09:57:51Z)
• Text-guided 3D Human Motion Generation with Keyframe-based Parallel Skip Transformer [62.29951737214263]
  Existing algorithms directly generate the full sequence which is expensive and prone to errors. We propose KeyMotion, that generates plausible human motion sequences corresponding to input text. We use a Variationalcoder (VAE) with Kullback-Leibler regularization to project the Autoencoder into a latent space. For the reverse diffusion, we propose a novel Parallel Skip Transformer that performs cross-modal attention between the design latents and text condition.
  arXiv  Detail & Related papers  (2024-05-24T11:12:37Z)
• AIris: An AI-powered Wearable Assistive Device for the Visually Impaired [0.0]
  We introduce AIris, an AI-powered wearable device that provides environmental awareness and interaction capabilities to visually impaired users. We have created a functional prototype system that operates effectively in real-world conditions.
  arXiv  Detail & Related papers  (2024-05-13T10:09:37Z)
• Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference [5.711221299998126]
  Persona-Gestor is a novel end-to-end generative model designed to generate highly personalized 3D full-body gestures. The model combines a fuzzy feature extractor and a non-autoregressive Adaptive Layer Normalization (AdaLN) transformer diffusion architecture. Persona-Gestor improves the system's usability and generalization capabilities.
  arXiv  Detail & Related papers  (2024-03-16T04:40:10Z)
• Prompt-to-OS (P2OS): Revolutionizing Operating Systems and Human-Computer Interaction with Integrated AI Generative Models [10.892991111926573]
  We present a paradigm for human-computer interaction that revolutionizes the traditional notion of an operating system. Within this innovative framework, user requests issued to the machine are handled by an interconnected ecosystem of generative AI models. This visionary concept raises significant challenges, including privacy, security, trustability, and the ethical use of generative models.
  arXiv  Detail & Related papers  (2023-10-07T17:16:34Z)
• "No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy [70.45420918526926]
  We present LILAC, a framework for incorporating and adapting to natural language corrections online during execution. Instead of discrete turn-taking between a human and robot, LILAC splits agency between the human and robot. We show that our corrections-aware approach obtains higher task completion rates, and is subjectively preferred by users.
  arXiv  Detail & Related papers  (2023-01-06T15:03:27Z)
• Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages [58.43299730989809]
  We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data. We induce a pseudo language as a compact discrete representation, and formulate a self-supervised pseudo speech recognition task. This process stands on its own, or can be applied as low-cost second-stage pre-training.
  arXiv  Detail & Related papers  (2022-05-02T17:59:02Z)
• Emotion-Aware Transformer Encoder for Empathetic Dialogue Generation [6.557082555839738]
  We propose an emotion-aware transformer encoder for capturing the emotional quotient in the user utterance. An emotion detector module determines the affective state of the user in the initial phase. A novel transformer encoder is proposed that adds and normalizes the word embedding with emotion embedding.
  arXiv  Detail & Related papers  (2022-04-24T17:05:36Z)
• Multimodal Emotion Recognition using Transfer Learning from Speaker Recognition and BERT-based models [53.31917090073727]
  We propose a neural network-based emotion recognition framework that uses a late fusion of transfer-learned and fine-tuned models from speech and text modalities. We evaluate the effectiveness of our proposed multimodal approach on the interactive emotional dyadic motion capture dataset.
  arXiv  Detail & Related papers  (2022-02-16T00:23:42Z)
• Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning [52.73083137245969]
  We present a generative adversarial network to synthesize 3D pose sequences of co-speech upper-body gestures with appropriate affective expressions. Our network consists of two components: a generator to synthesize gestures from a joint embedding space of features encoded from the input speech and the seed poses, and a discriminator to distinguish between the synthesized pose sequences and real 3D pose sequences.
  arXiv  Detail & Related papers  (2021-07-31T15:13:39Z)
• Relaxed Attention: A Simple Method to Boost Performance of End-to-End Automatic Speech Recognition [27.530537066239116]
  We introduce the concept of relaxed attention, which is a gradual injection of a uniform distribution to the encoder-decoder attention weights during training. We find that transformers trained with relaxed attention outperform the standard baseline models consistently during decoding with external language models. On WSJ, we set a new benchmark for transformer-based end-to-end speech recognition with a word error rate of 3.65%, outperforming state of the art (4.20%) by 13.1% relative.
  arXiv  Detail & Related papers  (2021-07-02T21:01:17Z)
• VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputs [103.99315770490163]
  We present a framework for text generation from multimodal inputs consisting of video plus text, speech, or audio. Experiments demonstrate that our approach based on a single architecture outperforms the state-of-the-art on three video-based text-generation tasks.
  arXiv  Detail & Related papers  (2021-01-28T15:22:36Z)
• SG-Net: Syntax Guided Transformer for Language Representation [58.35672033887343]
  We propose using syntax to guide the text modeling by incorporating explicit syntactic constraints into attention mechanisms for better linguistically motivated word representations. In detail, for self-attention network (SAN) sponsored Transformer-based encoder, we introduce syntactic dependency of interest (SDOI) design into the SAN to form an SDOI-SAN with syntax-guided self-attention. Experiments on popular benchmark tasks, including machine reading comprehension, natural language inference, and neural machine translation show the effectiveness of the proposed SG-Net design.
  arXiv  Detail & Related papers  (2020-12-27T11:09:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.