VoiceSHIELD-Small: Real-Time Malicious Speech Detection and Transcription

• URL: http://arxiv.org/abs/2603.07708v1
• Date: Sun, 08 Mar 2026 16:23:42 GMT
• Title: VoiceSHIELD-Small: Real-Time Malicious Speech Detection and Transcription
• Abstract summary: This paper introduces VoiceSHIELD-Small, a lightweight model that works in real time.<n>It can transcribe speech and detect whether it is safe or harmful, all in one step.<n>Built on OpenAI's Whisper-small encoder, VoiceSHIELD adds a mean-pooling layer and a simple classification head.
• Score: 2.0225826789157404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Voice interfaces are quickly becoming a common way for people to interact with AI systems. This also brings new security risks, such as prompt injection, social engineering, and harmful voice commands. Traditional security methods rely on converting speech to text and then filtering that text, which introduces delays and can ignore important audio cues. This paper introduces VoiceSHIELD-Small, a lightweight model that works in real time. It can transcribe speech and detect whether it is safe or harmful, all in one step. Built on OpenAI's Whisper-small encoder, VoiceSHIELD adds a mean-pooling layer and a simple classification head. It takes just 90-120 milliseconds to classify audio on mid-tier GPUs, while transcription happens at the same time. Tested on a balanced set of 947 audio clips, the model achieved 99.16 percent accuracy and an F1 score of 0.9865. At the default setting, it missed 2.33 percent of harmful inputs. Cross-validation showed consistent performance (F1 standard deviation = 0.0026). The paper also covers the model's design, training data, performance trade-offs, and responsible use guidelines. VoiceSHIELD is released under the MIT license to encourage further research and adoption in voice AI security.

Related papers

• ARENA: Automated Red-Teaming for Large Audio Language Models [30.804646851033567]
  We study automated audio-grounded red-teaming, where a text query must remain safe in isolation while the joint text-audio input induces harmful target behavior.<n>We propose ARENA, a closed-loop framework that trains a controller on an independent 2,000case text-audio dataset.<n>On 520 held-out AdvBench objectives, ARENA achieves FDR/PSR of 87.9/100.0%, 71.5/96.3%, 68.1/100.0%, and 75.4/98.5%.
  arXiv  Detail & Related papers  (2026-08-16T07:03:53Z)
• Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Models [56.91801348360746]
  Large audio-language models (LALMs) generalize across speech, sound, and music.<n> unified decoders can exhibit a emphtemporal smoothing bias<n>We propose emphTemporal Contrastive Decoding (TCD), a training-free decoding method for unified LALMs.
  arXiv  Detail & Related papers  (2026-04-16T02:30:41Z)
• Abjad AI at NADI 2025: CATT-Whisper: Multimodal Diacritic Restoration Using Text and Speech Representations [1.1391158217994781]
  We tackle the Diacritic Restoration (DR) task for Arabic dialectal sentences using a multimodal approach.<n>We propose a model that represents the text modality using an encoder extracted from our own pre-trained model named CATT.<n>Our experiments show that the proposed approach achieves a word error rate (WER) of 0.25 and a character error rate (CER) of 0.9 on the development set.
  arXiv  Detail & Related papers  (2025-10-28T09:58:18Z)
• When Good Sounds Go Adversarial: Jailbreaking Audio-Language Models with Benign Inputs [1.911526481015]
  Our research introduces WhisperInject, a two-stage adversarial audio attack framework.<n>It can manipulate state-of-the-art audio language models to generate harmful content.<n>Our method uses imperceptible perturbations in audio inputs that remain benign to human listeners.
  arXiv  Detail & Related papers  (2025-08-05T12:14:01Z)
• Exposing Synthetic Speech: Model Attribution and Detection of AI-generated Speech via Audio Fingerprints [11.703509488782345]
  We introduce a training-free, yet effective approach for detecting AI-generated speech.<n>We tackle three key tasks: (1) single-model attribution in an open-world setting, (2) multi-model attribution in a closed-world setting, and (3) detection of synthetic versus real speech.
  arXiv  Detail & Related papers  (2024-11-21T10:55:49Z)
• VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment [101.2489492032816]
  VALL-E R is a robust and efficient zero-shot Text-to-Speech system. This research has the potential to be applied to meaningful projects, including the creation of speech for those affected by aphasia.
  arXiv  Detail & Related papers  (2024-06-12T04:09:44Z)
• SuperVoice: Text-Independent Speaker Verification Using Ultrasound Energy in Human Speech [10.354590276508283]
  Voice-activated systems are integrated into a variety of desktop, mobile, and Internet-of-Things (IoT) devices. Existing speaker verification techniques distinguish individual speakers via the spectrographic features extracted from an audible frequency range of voice commands. We propose a speaker verification system, SUPERVOICE, that uses a two-stream architecture with a feature fusion mechanism to generate distinctive speaker models.
  arXiv  Detail & Related papers  (2022-05-28T18:00:50Z)
• Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data [145.95460945321253]
  We introduce two pre-training tasks for the encoder-decoder network using acoustic units, i.e., pseudo codes. The proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over the method without decoder pre-training.
  arXiv  Detail & Related papers  (2022-03-31T15:33:56Z)
• Self-supervised Learning with Random-projection Quantizer for Speech Recognition [51.24368930992091]
  We present a simple and effective self-supervised learning approach for speech recognition. The approach learns a model to predict masked speech signals, in the form of discrete labels. It achieves similar word-error-rates as previous work using self-supervised learning with non-streaming models.
  arXiv  Detail & Related papers  (2022-02-03T21:29:04Z)
• Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition [52.71604809100364]
  We propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets.
  arXiv  Detail & Related papers  (2021-10-11T00:08:48Z)
• Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narration [62.75234183218897]
  We propose a one-stage context-aware framework to generate natural and coherent target speech without any training data of the speaker. We generate the mel-spectrogram of the edited speech with a transformer-based decoder. It outperforms a recent zero-shot TTS engine by a large margin.
  arXiv  Detail & Related papers  (2021-09-12T04:17:53Z)
• Unsupervised Speech Recognition [55.864459085947345]
  wav2vec-U, short for wav2vec Unsupervised, is a method to train speech recognition models without any labeled data. We leverage self-supervised speech representations to segment unlabeled audio and learn a mapping from these representations to phonemes via adversarial training. On the larger English Librispeech benchmark, wav2vec-U achieves a word error rate of 5.9 on test-other, rivaling some of the best published systems trained on 960 hours of labeled data from only two years ago.
  arXiv  Detail & Related papers  (2021-05-24T04:10:47Z)
• Knowledge Transfer for Efficient On-device False Trigger Mitigation [17.53768388104929]
  An undirected utterance is termed as a "false trigger" and false trigger mitigation (FTM) is essential for designing a privacy-centric smart assistant. We propose an LSTM-based FTM architecture which determines the user intent from acoustic features directly without explicitly generating ASR transcripts.
  arXiv  Detail & Related papers  (2020-10-20T20:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.