Implementation Of Tiny Machine Learning Models On Arduino 33 BLE For Gesture And Speech Recognition

• URL: http://arxiv.org/abs/2207.12866v1
• Date: Sat, 23 Jul 2022 10:53:26 GMT
• Title: Implementation Of Tiny Machine Learning Models On Arduino 33 BLE For Gesture And Speech Recognition
• Authors: Viswanatha V, Ramachandra A.C, Raghavendra Prasanna, Prem Chowdary Kakarla, Viveka Simha PJ, Nishant Mohan
• Abstract summary: Here in the implementation of hand gesture recognition, TinyML model is trained and deployed from EdgeImpulse framework for hand gesture recognition. In the implementation of speech recognition, TinyML model is trained and deployed from EdgeImpulse framework for speech recognition. Arduino Nano 33 BLE device having built-in microphone can make an RGB LED glow like red, green or blue based on keyword pronounced.
• Score: 6.8324958655038195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this article gesture recognition and speech recognition applications are implemented on embedded systems with Tiny Machine Learning (TinyML). It features 3-axis accelerometer, 3-axis gyroscope and 3-axis magnetometer. The gesture recognition,provides an innovative approach nonverbal communication. It has wide applications in human-computer interaction and sign language. Here in the implementation of hand gesture recognition, TinyML model is trained and deployed from EdgeImpulse framework for hand gesture recognition and based on the hand movements, Arduino Nano 33 BLE device having 6-axis IMU can find out the direction of movement of hand. The Speech is a mode of communication. Speech recognition is a way by which the statements or commands of human speech is understood by the computer which reacts accordingly. The main aim of speech recognition is to achieve communication between man and machine. Here in the implementation of speech recognition, TinyML model is trained and deployed from EdgeImpulse framework for speech recognition and based on the keywords pronounced by human, Arduino Nano 33 BLE device having built-in microphone can make an RGB LED glow like red, green or blue based on keyword pronounced. The results of each application are obtained and listed in the results section and given the analysis upon the results.

Related papers

• RoboOmni: Proactive Robot Manipulation in Omni-modal Context [165.09049429566238]
  We introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands.<n>We present RoboOmni, a framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution.<n>Experiments in simulation and real-world settings show that Robo Omni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.
  arXiv  Detail & Related papers  (2025-10-27T18:49:03Z)
• A Surveillance Based Interactive Robot [0.0]
  We build a mobile surveillance robot that streams video in real time and responds to speech so a user can monitor and steer it from a phone or browser.<n>The system uses two Raspberry Pi 4 units: a front unit on a differential drive base with camera, mic, and speaker, and a central unit that serves the live feed and runs perception.<n>For voice interaction, we use Python libraries for speech recognition, multilingual translation, and text-to-speech.
  arXiv  Detail & Related papers  (2025-08-18T19:09:43Z)
• Understanding Co-speech Gestures in-the-wild [52.5993021523165]
  We introduce a new framework for co-speech gesture understanding in the wild. We propose three new tasks and benchmarks to evaluate a model's capability to comprehend gesture-text-speech associations. We present a new approach that learns a tri-modal speech-text-video-gesture representation to solve these tasks.
  arXiv  Detail & Related papers  (2025-03-28T17:55:52Z)
• FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs [63.8261207950923]
  FunAudioLLM is a model family designed to enhance natural voice interactions between humans and large language models (LLMs) At its core are two innovative models: SenseVoice, which handles multilingual speech recognition, emotion recognition, and audio event detection; and CosyVoice, which facilitates natural speech generation with control over multiple languages, timbre, speaking style, and speaker identity. The models related to SenseVoice and CosyVoice have been open-sourced on Modelscope and Huggingface, along with the corresponding training, inference, and fine-tuning codes released on GitHub.
  arXiv  Detail & Related papers  (2024-07-04T16:49:02Z)
• Automatic Speech Recognition for Hindi [0.6292138336765964]
  The research involved developing a web application and designing a web interface for speech recognition. The web application manages large volumes of audio files and their transcriptions, facilitating human correction of ASR transcripts. The web interface for speech recognition records 16 kHz mono audio from any device running the web app, performs voice activity detection (VAD), and sends the audio to the recognition engine.
  arXiv  Detail & Related papers  (2024-06-26T07:39:20Z)
• ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis [50.69464138626748]
  We present ConvoFusion, a diffusion-based approach for multi-modal gesture synthesis. Our method proposes two guidance objectives that allow the users to modulate the impact of different conditioning modalities. Our method is versatile in that it can be trained either for generating monologue gestures or even the conversational gestures.
  arXiv  Detail & Related papers  (2024-03-26T17:59:52Z)
• On decoder-only architecture for speech-to-text and large language model integration [59.49886892602309]
  Speech-LLaMA is a novel approach that effectively incorporates acoustic information into text-based large language models. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement over strong baselines.
  arXiv  Detail & Related papers  (2023-07-08T06:47:58Z)
• Plug-and-Play Multilingual Few-shot Spoken Words Recognition [3.591566487849146]
  We propose PLiX, a multilingual and plug-and-play keyword spotting system. Our few-shot deep models are learned with millions of one-second audio clips across 20 languages. We show that PLiX can generalize to novel spoken words given as few as just one support example.
  arXiv  Detail & Related papers  (2023-05-03T18:58:14Z)
• VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning [119.49605266839053]
  We propose a unified cross-modal representation learning framework VATLM (Visual-Audio-Text Language Model) The proposed VATLM employs a unified backbone network to model the modality-independent information. In order to integrate these three modalities into one shared semantic space, VATLM is optimized with a masked prediction task of unified tokens.
  arXiv  Detail & Related papers  (2022-11-21T09:10:10Z)
• Robotic Detection of a Human-Comprehensible Gestural Language for Underwater Multi-Human-Robot Collaboration [16.823029377470363]
  We present a motion-based robotic communication framework that enables non-verbal communication among autonomous underwater vehicles (AUVs) and human divers. We design a gestural language for AUV-to-A communication which can be easily understood by divers observing the conversation. To allow As to visually understand a gesture from another AUV, we propose a deep network (RRCommNet) which exploits a self-attention mechanism to learn to recognize each message by extracting discrimi-temporal features.
  arXiv  Detail & Related papers  (2022-07-12T06:04:12Z)
• Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion [89.01668641930206]
  We present a framework for modeling interactional communication in dyadic conversations. We autoregressively output multiple possibilities of corresponding listener motion. Our method organically captures the multimodal and non-deterministic nature of nonverbal dyadic interactions.
  arXiv  Detail & Related papers  (2022-04-18T17:58:04Z)
• Passing a Non-verbal Turing Test: Evaluating Gesture Animations Generated from Speech [6.445605125467574]
  In this paper, we propose a novel, data-driven technique for generating gestures directly from speech. Our approach is based on the application of Generative Adversarial Neural Networks (GANs) to model the correlation rather than causation between speech and gestures. For the study, we animate the generated gestures on a virtual character. We find that users are not able to distinguish between the generated and the recorded gestures.
  arXiv  Detail & Related papers  (2021-07-01T19:38:43Z)
• Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
  Sign language is used by deaf or speech impaired people to communicate. Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance. Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
  arXiv  Detail & Related papers  (2021-03-16T03:38:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.