SignMimic: Robust High-Quality Sign Language Motion Generation via Human-Shape-Oblivious Pose Transfer Guidance

• URL: http://arxiv.org/abs/2609.14122v1
• Date: Sat, 12 Sep 2026 19:57:16 GMT
• Title: SignMimic: Robust High-Quality Sign Language Motion Generation via Human-Shape-Oblivious Pose Transfer Guidance
• Abstract summary: We study the challenge of sign language mimicking: given a driving and a single reference frame, synthesize a video where the target signer reproduces the source motion.<n>Prior pipelines entangle rigid motion, non-rigid deformation, and view-dependent completion in a monolithic generator.<n>We present Signimic, which (i) applies a TNet-based model to study SE(3) rigid canonicalization to stabilize global pose, (ii) performs non-rigid adaptation in a canonical space to preserve fine-grained articulators (hands/face) and coarticulation via NIF2D, and (iii)
• Score: 7.47809638392782
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the challenge of sign language video mimicking: given a driving video and a single reference frame, synthesize a video where the target signer reproduces the source motion while preserving identity and linguistic form. Prior pipelines entangle rigid motion, non-rigid deformation, and view-dependent completion in a monolithic generator, causing handshape drift and spatio-temporal instability. We present SignMimic, which (i) applies a TNet-based model to study SE(3) rigid canonicalization to stabilize global pose, (ii) performs non-rigid adaptation in a canonical space to preserve fine-grained articulators (hands/face) and coarticulation via NIF2D, and (iii) uses Pose-MAE-style completion before conditional video diffusion. This factorization injects geometric and linguistic priors, yielding shape and spatio-temporal consistency. On several large-scale datasets (ASL 50K, How2Sign, CSL News), SignMimic achieves state-of-the-art-level performance on video quality, identity similarity, and frame continuity while also achieving minimal loss when performing back translation (SLT) on generated videos. Ablations confirm the role of rigid canonicalization, non-rigid adaptation, and completion. Code, model checkpoints, and video examples will be released.

Related papers

• RASA: Disentangled Spatial-Motional Priors for Cross-Identity Character Animation [85.41886214825894]
  Cross-identity character animation aims to drive a target identity from a reference image to follow the motion of a source character from a driving video.<n>We introduce Reference-Aware Structural Alignment (RASA), a framework that disentangles spatial mapping from motion control.<n>Our work establishes a new paradigm showing disentangled spatial and motional priors are key to robust character animation.
  arXiv  Detail & Related papers  (2026-08-28T11:36:44Z)
• SignNet-1M: Large-Scale Multilingual Sign Language Video Dataset with Downstream Benchmarks [41.59280653082095]
  SignNet-1M is a large-scale augmented dataset spanning ASL, CSL, and German Sign Language (DGS)<n>It synthesizes realistic variations along three axes: novel-view rendering (rotation and zoom), scene/identity editing, and post-rendering augmentations.<n>We show that training with SignNet-1M consistently improves generalization under cross-view, cross-background, cross-identity, and post-rendering shifts.
  arXiv  Detail & Related papers  (2026-06-23T09:51:03Z)
• From Synchrony to Sequence: Exo-to-Ego Generation via Interpolation [54.68405211129937]
  Exo-Seqto-Ego video generation aims to synthesize a first-person view from a synchronized third-person and corresponding camera poses.<n> synchronized exo-ego data inherently introduces substantial-temporal discontinuities, violating the smooth-motion assumptions of standard video generation benchmarks.<n>We propose Syn2Ex-Forcing, a sequential sequence model that interpolates between the source and target videos to form a single continuous signal.
  arXiv  Detail & Related papers  (2026-04-15T12:32:25Z)
• SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing [76.349958946335]
  We present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling.<n>First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames.<n>Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks.
  arXiv  Detail & Related papers  (2026-03-19T17:59:51Z)
• IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animation [58.297199313494]
  Implicit methods capture motion semantics directly from driving video, but suffer from identity leakage and entanglement between motion and appearance.<n>We propose a novel implicit motion representation that compresses per-frame motion into compact 1D motion tokens.<n>Our methodology employs a three-stage training strategy to enhance the training efficiency and ensure high fidelity.
  arXiv  Detail & Related papers  (2026-02-07T11:17:20Z)
• Motion is the Choreographer: Learning Latent Pose Dynamics for Seamless Sign Language Generation [24.324964949728045]
  We propose a new paradigm for sign language video generation that decouples motion semantics from signer identity.<n>First, we construct a signer-independent multimodal motion lexicon, where each gloss is stored as identity-agnostic pose, gesture, and 3D mesh sequences.<n>This compact representation enables our second key innovation: a discrete-to-continuous motion synthesis stage that transforms retrieved gloss sequences into temporally coherent motion trajectories.
  arXiv  Detail & Related papers  (2025-08-06T03:23:10Z)
• Beyond Gloss: A Hand-Centric Framework for Gloss-Free Sign Language Translation [18.544615425772903]
  Sign Language Translation (SLT) is a challenging task that requires bridging the modality gap between visual and linguistic information.<n>We introduce textbfBeyondGloss, a novel gloss-free SLT framework that leverages thetemporal-aware reasoning capabilities of VideoLLMs.<n>Beyondtexttext achieves state-of-the-art performance on the PhoenixT14 and CSL-Daily benchmarks, demonstrating the effectiveness of the proposed framework.
  arXiv  Detail & Related papers  (2025-07-31T14:06:07Z)
• Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization [20.063863466319326]
  SignViP is a novel framework that incorporates multiple fine-grained conditions for improved generation fidelity.<n>SignViP achieves state-of-the-art performance across metrics, including video quality temporal coherence, and semantic fidelity.
  arXiv  Detail & Related papers  (2025-06-19T02:56:06Z)
• Motion-Aware Concept Alignment for Consistent Video Editing [57.08108545219043]
  We introduce MoCA-Video (Motion-Aware Concept Alignment in Video), a training-free framework bridging the gap between image-domain semantic mixing and video.<n>Given a generated video and a user-provided reference image, MoCA-Video injects the semantic features of the reference image into a specific object within the video.<n>We evaluate MoCA's performance using the standard SSIM, image-level LPIPS, temporal LPIPS, and introduce a novel metric CASS (Conceptual Alignment Shift Score) to evaluate the consistency and effectiveness of the visual shifts between the source prompt and the modified video frames
  arXiv  Detail & Related papers  (2025-06-01T13:28:04Z)
• Pose-Guided Fine-Grained Sign Language Video Generation [18.167413937989867]
  We propose a novel Pose-Guided Motion Model (PGMM) for generating fine-grained and motion-consistent sign language videos. Firstly, we propose a new Coarse Motion Module (CMM), which completes the deformation of features by optical flow warping. Secondly, we propose a new Pose Fusion Module (PFM), which guides the modal fusion of RGB and pose features.
  arXiv  Detail & Related papers  (2024-09-25T07:54:53Z)
• Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment [130.15775113897553]
  Finsta is a fine-grained structural-temporal alignment learning method. It consistently improves the existing 13 strong-tuning video-language models.
  arXiv  Detail & Related papers  (2024-06-27T15:23:36Z)
• LEO: Generative Latent Image Animator for Human Video Synthesis [38.99490968487773]
  We propose a novel framework for human video synthesis, placing emphasis on synthesizing-temporal coherency. Our key idea is to represent motion as a sequence of flow maps in the generation process, which inherently isolate motion from appearance. We implement this idea via a flow-based image animator and a Latent Motion Diffusion Model (LMDM)
  arXiv  Detail & Related papers  (2023-05-06T09:29:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.