Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models

• URL: http://arxiv.org/abs/2607.11801v1
• Date: Mon, 13 Jul 2026 16:53:08 GMT
• Title: Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models
• Abstract summary: We introduce IAAN, a training-free and label-free method that scores each feed-forward neuron in the audio encoder.<n>It amplifies a small set of the highest-scoring neurons at inference.<n>It improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5- Omni, and 9.7 on Kimi-Audio.
• Score: 47.620410539694774
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large audio-language models (LALMs) often underperform on fine-grained, non-semantic attributes of speech, such as a speaker's emotion, despite strong performance on speech content. Improving this without the cost of retraining calls for an effective inference-time intervention, yet most existing methods intervene only after the audio encoder and operate at a relatively coarse granularity. The encoder itself, where acoustic information is first extracted from the waveform, remains largely unexplored, especially at the level of individual neurons. We introduce IAAN, Identifying and Amplifying Acoustic Neurons, a training-free and label-free method that scores each feed-forward neuron in the audio encoder by contrasting its activation on the real waveform with that on a noise reference lacking the real audio's acoustic information. IAAN then amplifies a small set of the highest-scoring neurons at inference. Across ten non-semantic speech attributes, IAAN improves average accuracy by 25.7 points on Audio-Flamingo-3, 21.4 on Qwen2.5-Omni, and 9.7 on Kimi-Audio. It also improves a model already explicitly fine-tuned to prioritize acoustic evidence. In controlled comparisons, both the encoder locus and neuron-level selectivity prove necessary for this gain. Intervening after the encoder, at the decoding side or inside the language model, yields little to no improvement, or even deteriorates accuracy. The improvement also depends on which specific neurons are amplified, not merely on their number, confirming that IAAN's acoustic score succeeds in identifying the neurons that matter. These results show that a small, precisely targeted intervention inside the audio encoder is an effective and largely untapped way to strengthen the acoustic understanding of LALMs, opening a new direction for inference-time methods that improve acoustic perception through neuron-level access to the encoder.

Related papers

• Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition [58.25449304752214]
  We study whether explicit acoustic cues are grounded when the raw audio is already available.<n>We deriving six interpretable acoustic concept tokens from the standardised eGeMAPS paralinguistic feature set.<n>Aligned tokens improve unweighted average recall (UAR), whereas shuffled, conflicting, or corrupted tokens reduce performance.<n>We argue that token-only interventions provide a practical way to probe audio-grounded cue use, robustness, and interpretability in ALM-based affective computing.
  arXiv  Detail & Related papers  (2026-06-05T14:26:06Z)
• SonicBench: Dissecting the Physical Perception Bottleneck in Large Audio Language Models [30.62556746827114]
  Large Audio Language Models (LALMs) excel at semantic and paralinguistic tasks, yet their ability to perceive the fundamental physical attributes of audio remains under-explored.<n>We introduce SonicBench, a psychophysically grounded benchmark that systematically evaluates 12 core physical attributes across five dimensions.<n>Our evaluation reveals a substantial deficiency in LALMs' foundational auditory understanding.
  arXiv  Detail & Related papers  (2026-01-16T07:10:57Z)
• Semantic Codebooks as Effective Priors for Neural Speech Compression [3.4074476957610074]
  SemDAC is a semantic-aware neural audio that leverages semantic codebooks as effective priors for speech compression.<n>A FiLM-conditioned decoder reconstructs audio conditioned on the semantic tokens, improving efficiency in the use of acoustic codebooks.
  arXiv  Detail & Related papers  (2025-12-25T12:49:41Z)
• Decoding Phone Pairs from MEG Signals Across Speech Modalities [0.4054486015338004]
  We investigated magnetoencephalography signals to decode phones from brain activity during speech production and perception tasks.<n>Our results demonstrate significantly higher decoding accuracy during speech production compared to passive listening and playback modalities.
  arXiv  Detail & Related papers  (2025-05-21T10:31:34Z)
• Multi-class Decoding of Attended Speaker Direction Using Electroencephalogram and Audio Spatial Spectrum [13.036563238499026]
  Decoding the directional focus of an attended speaker from listeners' electroencephalogram (EEG) signals is essential for developing brain-computer interfaces.<n>By integrating audio spatial spectra with EEG features, the decoding accuracy can be effectively improved.<n>The proposed Sp-EEG-Deformer model achieves notable 14-class decoding accuracies of 55.35% and 57.19% in leave-one-subject-out and leave-one-trial-out scenarios.
  arXiv  Detail & Related papers  (2024-11-11T12:32:26Z)
• End-to-End Binaural Speech Synthesis [71.1869877389535]
  We present an end-to-end speech synthesis system that combines a low-bitrate audio system with a powerful decoder. We demonstrate the capability of the adversarial loss in capturing environment effects needed to create an authentic auditory scene.
  arXiv  Detail & Related papers  (2022-07-08T05:18:36Z)
• Self-supervised models of audio effectively explain human cortical responses to speech [71.57870452667369]
  We capitalize on the progress of self-supervised speech representation learning to create new state-of-the-art models of the human auditory system. We show that these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.
  arXiv  Detail & Related papers  (2022-05-27T22:04:02Z)
• Exploiting Cross Domain Acoustic-to-articulatory Inverted Features For Disordered Speech Recognition [57.15942628305797]
  Articulatory features are invariant to acoustic signal distortion and have been successfully incorporated into automatic speech recognition systems for normal speech. This paper presents a cross-domain acoustic-to-articulatory (A2A) inversion approach that utilizes the parallel acoustic-articulatory data of the 15-hour TORGO corpus in model training. Cross-domain adapted to the 102.7-hour UASpeech corpus and to produce articulatory features.
  arXiv  Detail & Related papers  (2022-03-19T08:47:18Z)
• Data Augmentation based Consistency Contrastive Pre-training for Automatic Speech Recognition [18.303072203996347]
  Self-supervised acoustic pre-training has achieved amazing results on the automatic speech recognition (ASR) task. Most of the successful acoustic pre-training methods use contrastive learning to learn the acoustic representations. In this letter, we design a novel consistency contrastive learning (CCL) method by utilizing data augmentation for acoustic pre-training.
  arXiv  Detail & Related papers  (2021-12-23T13:23:17Z)
• Deliberation Model Based Two-Pass End-to-End Speech Recognition [52.45841282906516]
  A two-pass model has been proposed to rescore streamed hypotheses using the non-streaming Listen, Attend and Spell (LAS) model. The model attends to acoustics to rescore hypotheses, as opposed to a class of neural correction models that use only first-pass text hypotheses. A bidirectional encoder is used to extract context information from first-pass hypotheses.
  arXiv  Detail & Related papers  (2020-03-17T22:01:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.