Making Acoustic Side-Channel Attacks on Noisy Keyboards Viable with LLM-Assisted Spectrograms' "Typo" Correction

• URL: http://arxiv.org/abs/2504.11622v1
• Date: Tue, 15 Apr 2025 21:23:25 GMT
• Title: Making Acoustic Side-Channel Attacks on Noisy Keyboards Viable with LLM-Assisted Spectrograms' "Typo" Correction
• Authors: Seyyed Ali Ayati, Jin Hyun Park, Yichen Cai, Marcus Botacin,
• Abstract summary: Large integration of microphones into devices increases the opportunities for Acoustic Side-Channel Attacks (ASCAs)<n>Current State-Of-The-Art (SOTA) models for ASCAs exhibit limited robustness under realistic noisy conditions.<n>We present the first-of-its-kind approach that integrates Visual Transformers (VTs) and Large Language Models (LLMs) for ASCAs.
• Score: 5.0998111447316194
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The large integration of microphones into devices increases the opportunities for Acoustic Side-Channel Attacks (ASCAs), as these can be used to capture keystrokes' audio signals that might reveal sensitive information. However, the current State-Of-The-Art (SOTA) models for ASCAs, including Convolutional Neural Networks (CNNs) and hybrid models, such as CoAtNet, still exhibit limited robustness under realistic noisy conditions. Solving this problem requires either: (i) an increased model's capacity to infer contextual information from longer sequences, allowing the model to learn that an initially noisily typed word is the same as a futurely collected non-noisy word, or (ii) an approach to fix misidentified information from the contexts, as one does not type random words, but the ones that best fit the conversation context. In this paper, we demonstrate that both strategies are viable and complementary solutions for making ASCAs practical. We observed that no existing solution leverages advanced transformer architectures' power for these tasks and propose that: (i) Visual Transformers (VTs) are the candidate solutions for capturing long-term contextual information and (ii) transformer-powered Large Language Models (LLMs) are the candidate solutions to fix the ``typos'' (mispredictions) the model might make. Thus, we here present the first-of-its-kind approach that integrates VTs and LLMs for ASCAs. We first show that VTs achieve SOTA performance in classifying keystrokes when compared to the previous CNN benchmark. Second, we demonstrate that LLMs can mitigate the impact of real-world noise. Evaluations on the natural sentences revealed that: (i) incorporating LLMs (e.g., GPT-4o) in our ASCA pipeline boosts the performance of error-correction tasks; and (ii) the comparable performance can be attained by a lightweight, fine-tuned smaller LLM (67 times smaller than GPT-4o), using...

Related papers

• Improving Acoustic Side-Channel Attacks on Keyboards Using Transformers and Large Language Models [1.1674893622721483]
  This study explores deep learning techniques to enhance the effectiveness and applicability of acoustic side-channel attacks (ASCAs)<n>We present substantial improvements over prior research, with the CoAtNet model achieving state-of-the-art performance.<n>A key advancement is the introduction of a noise mitigation method for real-world scenarios.
  arXiv  Detail & Related papers  (2025-02-13T21:33:57Z)
• How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario [72.02391485962127]
  Speech Self-Supervised Learning (SSL) models achieve impressive performance on Automatic Speech Recognition (ASR) In low-resource language ASR, they encounter the domain mismatch problem between pre-trained and low-resource languages. We extend a conventional efficient fine-tuning scheme based on the adapter to handle these issues.
  arXiv  Detail & Related papers  (2024-11-27T10:51:00Z)
• Intent Detection in the Age of LLMs [3.755082744150185]
  Intent detection is a critical component of task-oriented dialogue systems (TODS) Traditional approaches relied on computationally efficient supervised sentence transformer encoder models. The emergence of generative large language models (LLMs) with intrinsic world knowledge presents new opportunities to address these challenges.
  arXiv  Detail & Related papers  (2024-10-02T15:01:55Z)
• Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models [42.891427362223176]
  Large language models (LLMs) based on decoder-only transformers have demonstrated superior text understanding capabilities.<n>We propose a novel framework to fully harness the capabilities of LLMs.<n>We further design an LLM-Infused Diffusion Transformer (LI-DiT) based on the framework.
  arXiv  Detail & Related papers  (2024-06-17T17:59:43Z)
• Advancing the Robustness of Large Language Models through Self-Denoised Smoothing [50.54276872204319]
  Large language models (LLMs) have achieved significant success, but their vulnerability to adversarial perturbations has raised considerable concerns. We propose to leverage the multitasking nature of LLMs to first denoise the noisy inputs and then to make predictions based on these denoised versions. Unlike previous denoised smoothing techniques in computer vision, which require training a separate model to enhance the robustness of LLMs, our method offers significantly better efficiency and flexibility.
  arXiv  Detail & Related papers  (2024-04-18T15:47:00Z)
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning [67.0609518552321]
  We propose to conduct Machine Vision Therapy which aims to rectify the noisy predictions from vision models. By fine-tuning with the denoised labels, the learning model performance can be boosted in an unsupervised manner.
  arXiv  Detail & Related papers  (2023-12-05T07:29:14Z)
• Modality Confidence Aware Training for Robust End-to-End Spoken Language Understanding [18.616202196061966]
  End-to-end (E2E) spoken language understanding (SLU) systems that generate a semantic parse from speech have become more promising recently. This approach uses a single model that utilizes audio and text representations from pre-trained speech recognition models (ASR) We propose a novel E2E SLU system that enhances robustness to ASR errors by fusing audio and text representations based on the estimated modality confidence of ASR hypotheses.
  arXiv  Detail & Related papers  (2023-07-22T17:47:31Z)
• Deliberation Model for On-Device Spoken Language Understanding [69.5587671262691]
  We propose a novel deliberation-based approach to end-to-end (E2E) spoken language understanding (SLU) We show that our approach can significantly reduce the degradation when moving from natural speech to synthetic speech training.
  arXiv  Detail & Related papers  (2022-04-04T23:48:01Z)
• RNN Transducer Models For Spoken Language Understanding [49.07149742835825]
  We show how RNN-T SLU models can be developed starting from pre-trained automatic speech recognition systems. In settings where real audio data is not available, artificially synthesized speech is used to successfully adapt various SLU models.
  arXiv  Detail & Related papers  (2021-04-08T15:35:22Z)
• Meta learning to classify intent and slot labels with noisy few shot examples [11.835266162072486]
  Spoken language understanding (SLU) models are notorious for being data-hungry. We propose a new SLU benchmarking task: few-shot robust SLU, where SLU comprises two core problems, intent classification (IC) and slot labeling (SL) We show the model consistently outperforms the conventional fine-tuning baseline and another popular meta-learning method, Model-Agnostic Meta-Learning (MAML), in terms of achieving better IC accuracy and SL F1.
  arXiv  Detail & Related papers  (2020-11-30T18:53:30Z)
• Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding [19.105304214638075]
  We introduce a novel framework for learning spoken language understanding. The framework consists of a conversational language modeling (CLM) pre-training task and a light encoder architecture. With the framework, we match the performance of state-of-the-art SLU results on Alexa internal datasets and on two public ones, adding only 4.4% parameters per task.
  arXiv  Detail & Related papers  (2020-10-09T03:53:37Z)
• Pretraining Techniques for Sequence-to-Sequence Voice Conversion [57.65753150356411]
  Sequence-to-sequence (seq2seq) voice conversion (VC) models are attractive owing to their ability to convert prosody. We propose to transfer knowledge from other speech processing tasks where large-scale corpora are easily available, typically text-to-speech (TTS) and automatic speech recognition (ASR) We argue that VC models with such pretrained ASR or TTS model parameters can generate effective hidden representations for high-fidelity, highly intelligible converted speech.
  arXiv  Detail & Related papers  (2020-08-07T11:02:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.