Impact Analysis of Speech Representation Learning Models for Acoustic Side-Channel Attack

• URL: http://arxiv.org/abs/2606.21210v2
• Date: Fri, 26 Jun 2026 13:32:46 GMT
• Title: Impact Analysis of Speech Representation Learning Models for Acoustic Side-Channel Attack
• Abstract summary: KEYAC is a dataset designed to analyze representation generalization for ASCA under both standard and VoIP settings.<n>We evaluate six representation learning models under zero-shot and partial fine-tuning settings using fully connected and convolutional networks.<n>Results show that while partial fine-tuning improves performance, models struggle to generalize across VoIP codecs.
• Score: 7.2972297703292135
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Acoustic side-channel attacks (ASCA) on keyboards have gained increasing attention, yet impact of speech representation learning models in ASCA remains unexplored. Addressing this, we introduce KEYAC, a dataset designed to analyze representation generalization for ASCA under both standard and VoIP codec settings. On KEYAC, we evaluate six representation learning models under zero-shot and partial fine-tuning settings using fully connected and convolutional networks. Results show that while partial fine-tuning improves performance, models struggle to generalize across VoIP codecs. We hypothesize this limitation stems from inadequate modeling of nonlinear feature interactions in conventional fine-tuning architectures. To address this, we employ Kolmogorov-Arnold Networks (KAN) for fine-tuning. Empirical results show that KAN-based fine-tuning consistently outperforms the baselines and establishes a new state-of-the-art on KEYAC.

Related papers

• RAE-AR: Taming Autoregressive Models with Representation Autoencoders [61.73674018219353]
  We introduce token simplification via distribution normalization to ease modeling difficulty and improve convergence.<n>We enhance prediction by incorporating Gaussian noise injection during training to mitigate exposure bias.<n>This work paves the way for a more unified architecture across visual understanding and generative modeling.
  arXiv  Detail & Related papers  (2026-04-02T02:39:28Z)
• StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models [98.72926158261937]
  We propose a training-free token pruning framework for Visual AutoRegressive models.<n>We employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information.<n>To maintain valid next-scale prediction under sparse tokens, we introduce a nearest neighbor feature propagation strategy.
  arXiv  Detail & Related papers  (2026-03-02T11:35:05Z)
• Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognition [51.03674130115878]
  We introduce the Knowledge-Informed Neural Network (KINN), a lightweight framework built upon a novel "compression-aggregation-compression" architecture.<n>KINN establishes a state-of-the-art in parameter-efficient recognition, offering exceptional generalization in data-scarce and out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-10-23T07:12:26Z)
• XLSR-Kanformer: A KAN-Intergrated model for Synthetic Speech Detection [1.8334165690218873]
  We propose a novel approach that replaces the traditional Multi-Layer Perceptron (MLP) in the XLSR-Conformer model with a Kolmogorov-Arnold Network (KAN)<n>Our experimental results demonstrate that the integration of KAN to XLSR-Conformer model can improve the performance by 60.55% relatively in Equal Error Rate (EER) LA and DF sets.
  arXiv  Detail & Related papers  (2025-10-08T06:58:58Z)
• Pushing the Performance of Synthetic Speech Detection with Kolmogorov-Arnold Networks and Self-Supervised Learning Models [1.7205106391379026]
  We propose a novel approach that replaces the traditional Multi-Layer Perceptron in the XLSR-Conformer model with a Kolmogorov-Arnold Network (KAN)<n>Our results show that integrating KAN into the SSL-based models can improve the performance by 60.55% relatively on LA and DF sets.
  arXiv  Detail & Related papers  (2025-06-17T03:30:58Z)
• $C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction [80.57232374640911]
  We propose a model-agnostic strategy called the Mask-And-Recover (MAR)<n>MAR integrates both inter- and intra-modality contextual correlations to enable global inference within extraction modules.<n>To better target challenging parts within each sample, we introduce a Fine-grained Confidence Score (FCS) model.
  arXiv  Detail & Related papers  (2025-04-01T13:01:30Z)
• Improving Acoustic Side-Channel Attacks on Keyboards Using Transformers and Large Language Models [1.1674893622721483]
  This study explores deep learning techniques to enhance the effectiveness and applicability of acoustic side-channel attacks (ASCAs)<n>We present substantial improvements over prior research, with the CoAtNet model achieving state-of-the-art performance.<n>A key advancement is the introduction of a noise mitigation method for real-world scenarios.
  arXiv  Detail & Related papers  (2025-02-13T21:33:57Z)
• EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance [4.345953264604522]
  We analyze and optimize the EnCLAP framework, a state-of-the-art model in automated audio captioning. We develop EnCLAP++, an enhanced version that significantly surpasses the original.
  arXiv  Detail & Related papers  (2024-09-02T12:23:18Z)
• You Only Train Once: A Unified Framework for Both Full-Reference and No-Reference Image Quality Assessment [45.62136459502005]
  We propose a network to perform full reference (FR) and no reference (NR) IQA. We first employ an encoder to extract multi-level features from input images. A Hierarchical Attention (HA) module is proposed as a universal adapter for both FR and NR inputs. A Semantic Distortion Aware (SDA) module is proposed to examine feature correlations between shallow and deep layers of the encoder.
  arXiv  Detail & Related papers  (2023-10-14T11:03:04Z)
• Adversarial Audio Synthesis with Complex-valued Polynomial Networks [60.231877895663956]
  Time-frequency (TF) representations in audio have been increasingly modeled real-valued networks. We introduce complex-valued networks called APOLLO, that integrate such complex-valued representations in a natural way. APOLLO results in $17.5%$ improvement over adversarial methods and $8.2%$ over the state-of-the-art diffusion models on SC09 in audio generation.
  arXiv  Detail & Related papers  (2022-06-14T12:58:59Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)
• Cross-scale Attention Model for Acoustic Event Classification [45.15898265162008]
  We propose a cross-scale attention (CSA) model, which explicitly integrates features from different scales to form the final representation. We show that the proposed CSA model can effectively improve the performance of current state-of-the-art deep learning algorithms.
  arXiv  Detail & Related papers  (2019-12-27T07:28:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.