OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis

• URL: http://arxiv.org/abs/2602.22949v1
• Date: Thu, 26 Feb 2026 12:41:24 GMT
• Title: OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis
• Authors: Junuk Cha, Jihyeon Kim, Han-Mu Park,
• Abstract summary: Fingerspelling recognition plays a crucial role in bridging the communication gap between Deaf and hearing communities.<n>We develop OpenFS, an open-source approach for fingerspelling recognition and synthesis.
• Score: 7.940141427569929
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fingerspelling is a component of sign languages in which words are spelled out letter by letter using specific hand poses. Automatic fingerspelling recognition plays a crucial role in bridging the communication gap between Deaf and hearing communities, yet it remains challenging due to the signing-hand ambiguity issue, the lack of appropriate training losses, and the out-of-vocabulary (OOV) problem. Prior fingerspelling recognition methods rely on explicit signing-hand detection, which often leads to recognition failures, and on a connectionist temporal classification (CTC) loss, which exhibits the peaky behavior problem. To address these issues, we develop OpenFS, an open-source approach for fingerspelling recognition and synthesis. We propose a multi-hand-capable fingerspelling recognizer that supports both single- and multi-hand inputs and performs implicit signing-hand detection by incorporating a dual-level positional encoding and a signing-hand focus (SF) loss. The SF loss encourages cross-attention to focus on the signing hand, enabling implicit signing-hand detection during recognition. Furthermore, without relying on the CTC loss, we introduce a monotonic alignment (MA) loss that enforces the output letter sequence to follow the temporal order of the input pose sequence through cross-attention regularization. In addition, we propose a frame-wise letter-conditioned generator that synthesizes realistic fingerspelling pose sequences for OOV words. This generator enables the construction of a new synthetic benchmark, called FSNeo. Through comprehensive experiments, we demonstrate that our approach achieves state-of-the-art performance in recognition and validate the effectiveness of the proposed recognizer and generator. Codes and data are available in: https://github.com/JunukCha/OpenFS.

Related papers

• Enhancing IMU-Based Online Handwriting Recognition via Contrastive Learning with Zero Inference Overhead [4.519836503888727]
  We propose a training framework designed to improve feature representation and recognition accuracy without increasing inference costs.<n>ECHWR utilizes a temporary auxiliary branch that aligns sensor signals with semantic text embeddings during the training phase.<n> Evaluations on the OnHW-Words500 dataset show that ECHWR significantly outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-02-04T13:44:54Z)
• Pose-Based Sign Language Spotting via an End-to-End Encoder Architecture [0.4083182125683813]
  We present a first step toward sign language retrieval by addressing the challenge of detecting the presence or absence of a query sign video.<n>Unlike conventional approaches that rely on intermediate gloss recognition or text-based matching, we propose an end-to-end model that directly operates on pose keypoints extracted from sign videos.<n>Our architecture employs an encoder-only backbone with a binary classification head to determine whether the query sign appears within the target sequence.
  arXiv  Detail & Related papers  (2025-12-09T15:49:23Z)
• AutoSign: Direct Pose-to-Text Translation for Continuous Sign Language Recognition [0.0]
  Continuously recognizing sign gestures and converting them to glosses plays a key role in bridging the gap between hearing and hearing-impaired communities.<n>We propose AutoSign, an autoregressive decoder-only transformer that directly translates pose sequences to natural language text.<n>By eliminating the multi-stage pipeline, AutoSign achieves substantial improvements on the Isharah-1000 dataset.
  arXiv  Detail & Related papers  (2025-07-26T07:28:33Z)
• Online Writer Retrieval with Chinese Handwritten Phrases: A Synergistic Temporal-Frequency Representation Learning Approach [53.189911918976655]
  We propose DOLPHIN, a novel retrieval model designed to enhance handwriting representations through synergistic temporal-frequency analysis.<n>We introduce OLIWER, a large-scale online writer retrieval dataset encompassing over 670,000 Chinese handwritten phrases from 1,731 individuals.<n>Our findings emphasize the significance of point sampling frequency and pressure features in improving handwriting representation quality.
  arXiv  Detail & Related papers  (2024-12-16T11:19:22Z)
• Provably Secure Disambiguating Neural Linguistic Steganography [66.30965740387047]
  The segmentation ambiguity problem, which arises when using language models based on subwords, leads to occasional decoding failures.<n>We propose a novel secure disambiguation method named SyncPool, which effectively addresses the segmentation ambiguity problem.<n> SyncPool does not change the size of the candidate pool or the distribution of tokens and thus is applicable to provably secure language steganography methods.
  arXiv  Detail & Related papers  (2024-03-26T09:25:57Z)
• Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information [67.78183175605761]
  Large Language Models are susceptible to adversarial prompt attacks. This vulnerability underscores a significant concern regarding the robustness and reliability of LLMs. We introduce a novel approach to detecting adversarial prompts at a token level.
  arXiv  Detail & Related papers  (2023-11-20T03:17:21Z)
• Open-Vocabulary Animal Keypoint Detection with Semantic-feature Matching [74.75284453828017]
  Open-Vocabulary Keypoint Detection (OVKD) task is innovatively designed to use text prompts for identifying arbitrary keypoints across any species. We have developed a novel framework named Open-Vocabulary Keypoint Detection with Semantic-feature Matching (KDSM) This framework combines vision and language models, creating an interplay between language features and local keypoint visual features.
  arXiv  Detail & Related papers  (2023-10-08T07:42:41Z)
• Weakly-supervised Fingerspelling Recognition in British Sign Language Videos [85.61513254261523]
  Previous fingerspelling recognition methods have not focused on British Sign Language (BSL) In contrast to previous methods, our method only uses weak annotations from subtitles for training. We propose a Transformer architecture adapted to this task, with a multiple-hypothesis CTC loss function to learn from alternative annotation possibilities.
  arXiv  Detail & Related papers  (2022-11-16T15:02:36Z)
• A Fine-Grained Visual Attention Approach for Fingerspelling Recognition in the Wild [17.8181080354116]
  Automatic recognition of fingerspelling can help resolve communication barriers when interacting with deaf people. Main challenges prevalent in fingerspelling recognition are the ambiguity in the gestures and strong articulation of the hands. We propose a fine-grained visual attention mechanism using the Transformer model for the sequence-to-sequence prediction task in the wild dataset.
  arXiv  Detail & Related papers  (2021-05-17T06:15:35Z)
• Fingerspelling Detection in American Sign Language [32.79935314131377]
  We consider the task of fingerspelling detection in raw, untrimmed sign language videos. This is an important step towards building real-world fingerspelling recognition systems. We propose a benchmark and a suite of evaluation metrics, some of which reflect the effect of detection on the downstream fingerspelling recognition task.
  arXiv  Detail & Related papers  (2021-04-03T02:11:09Z)
• Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
  Sign language is used by deaf or speech impaired people to communicate. Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance. Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
  arXiv  Detail & Related papers  (2021-03-16T03:38:17Z)
• Fast End-to-End Speech Recognition via a Non-Autoregressive Model and Cross-Modal Knowledge Transferring from BERT [72.93855288283059]
  We propose a non-autoregressive speech recognition model called LASO (Listen Attentively, and Spell Once) The model consists of an encoder, a decoder, and a position dependent summarizer (PDS)
  arXiv  Detail & Related papers  (2021-02-15T15:18:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.