Positional Encoding in the Context of Memristor-Based Analog Computation for Automatic Speech Recognition

• URL: http://arxiv.org/abs/2606.13379v1
• Date: Thu, 11 Jun 2026 14:05:58 GMT
• Title: Positional Encoding in the Context of Memristor-Based Analog Computation for Automatic Speech Recognition
• Abstract summary: We identify large output values of transformed positional encodings to cause major degradation within analog-to-digital conversion.<n>By adjusting the proportion of weight and precision bits of the ADC of specific memristor layers, we reduce the degradation of the execution by 50% relative.
• Score: 15.992009842323379
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Memristors provide a new chance for resource-efficient computation of neural models for natural language processing by enabling analog execution of vector-matrix-multiplication. Yet, computations on these devices are currently subject to larger distortion, both in weight programming and execution. In this work, we identify large output values of transformed positional encodings to cause major degradation within analog-to-digital conversion (ADC) as part of memristor-based computation. By adjusting the proportion of weight and precision bits of the ADC of specific memristor layers, we reduce the degradation of the execution by ~50% relative, while keeping the estimated energy consumption stable. Additionally, we investigate scenarios where the ADC cannot be modified. In that case the degradation can be reduced by ~30% relative after removing encoding-related linear transformations.

Related papers

• Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference [14.784763071210014]
  Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications.<n>We propose Reduced Matrix multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along their contraction dimensions.
  arXiv  Detail & Related papers  (2026-08-13T16:16:04Z)
• CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs [48.961770224590204]
  CODA is a GPU kernel abstraction that expresses these computations as GEMM-plus-epilogue programs.<n>We show that CODA offers a practical path toward combining framework-level productivity with hardware-level efficiency.
  arXiv  Detail & Related papers  (2026-05-19T02:30:43Z)
• STaMP: Sequence Transformation and Mixed Precision for Low-Precision Activation Quantization [21.93314755695813]
  Quantization is the key method for reducing inference latency, power and memory footprint of generative AI models.<n>We propose textitSequence Transformation and Mixed Precision (STaMP) quantization.
  arXiv  Detail & Related papers  (2025-10-30T17:53:42Z)
• Degeneracy Cutting: A Local and Efficient Post-Processing for Belief Propagation Decoding of Quantum Low-Density Parity-Check Codes [0.9558392439655014]
  Quantum low-density parity-check (qLDPC) codes are promising for realizing scalable fault-tolerant quantum computation.<n>A common approach to decoding qLDPC codes is to use the belief propagation (BP) decoder, followed by a post-processing step to enhance decoding accuracy.<n>For real-time decoding, the post-processing algorithm is desirable to have a small computational cost and rely only on local operations on the Tanner graph to facilitate parallel implementation.<n>We propose degeneracy cutting (DC), an efficient post-processing technique for the BP decoder that operates on information restricted to the support of each stabilizer
  arXiv  Detail & Related papers  (2025-10-09T18:01:30Z)
• Image Coding for Machines via Feature-Preserving Rate-Distortion Optimization [31.210700220124192]
  We show that an approach to reduce the effect of compression for a given task loss is to perform rate-distortion (RDO) using the distance between features.<n>We simplify the RDO formulation to make the distortion term computable using block-based encoders.<n>We show that simulations with transformed HEVC across multiple feature extractors and downstream networks show up to 17 % bit-rate savings for the same task accuracy.
  arXiv  Detail & Related papers  (2025-04-03T02:11:26Z)
• Accelerating Error Correction Code Transformers [56.75773430667148]
  We introduce a novel acceleration method for transformer-based decoders. We achieve a 90% compression ratio and reduce arithmetic operation energy consumption by at least 224 times on modern hardware.
  arXiv  Detail & Related papers  (2024-10-08T11:07:55Z)
• Intrinsic Voltage Offsets in Memcapacitive Bio-Membranes Enable High-Performance Physical Reservoir Computing [0.0]
  Reservoir computing is a brain-inspired machine learning framework for processing temporal data by mapping inputs into high-dimensional spaces. Here, we introduce a novel memcapacitor-based PRC that exploits internal voltage offsets to enable both monotonic and non-monotonic input-state correlations. Our approach and unprecedented performance are a major milestone towards high-performance full in-materia PRCs.
  arXiv  Detail & Related papers  (2024-04-27T05:47:38Z)
• Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference [12.371152982808914]
  We introduce the Adaptive Computation Module (ACM), a generic module that dynamically adapts its computational load to match the estimated difficulty of the input on a per-token basis.<n>An ACM consists of a sequence of learners that progressively refine the output of their preceding counterparts. An additional gating mechanism determines the optimal number of learners to execute for each token.<n>Our evaluation of transformer models in computer vision and speech recognition demonstrates that substituting layers with ACMs significantly reduces inference costs without degrading the downstream accuracy for a wide interval of user-defined budgets.
  arXiv  Detail & Related papers  (2023-12-15T20:39:43Z)
• Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs [75.40636935415601]
  Deep learning often faces the challenge of efficiently processing dynamic inputs, such as sensor data or user inputs. We take an incremental computing approach, looking to reuse calculations as the inputs change. We apply this approach to the transformers architecture, creating an efficient incremental inference algorithm with complexity proportional to the fraction of modified inputs.
  arXiv  Detail & Related papers  (2023-07-27T16:30:27Z)
• Compute Cost Amortized Transformer for Streaming ASR [23.950740806308687]
  We present a streaming, Transformer-based end-to-end automatic speech recognition architecture. Our architecture creates sparse computation pathways dynamically at inference time, resulting in selective use of compute resources throughout decoding. Our best model can achieve a 60% compute cost reduction with only a 3% relative word error rate (WER) increase.
  arXiv  Detail & Related papers  (2022-07-05T03:06:53Z)
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition [67.95996816744251]
  State-of-the-art language models (LMs) represented by long-short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming increasingly complex and expensive for practical applications. Current quantization methods are based on uniform precision and fail to account for the varying performance sensitivity at different parts of LMs to quantization errors. Novel mixed precision neural network LM quantization methods are proposed in this paper.
  arXiv  Detail & Related papers  (2021-11-29T12:24:02Z)
• Bayesian Transformer Language Models for Speech Recognition [59.235405107295655]
  State-of-the-art neural language models (LMs) represented by Transformers are highly complex. This paper proposes a full Bayesian learning framework for Transformer LM estimation.
  arXiv  Detail & Related papers  (2021-02-09T10:55:27Z)
• Predictive Coding Approximates Backprop along Arbitrary Computation Graphs [68.8204255655161]
  We develop a strategy to translate core machine learning architectures into their predictive coding equivalents. Our models perform equivalently to backprop on challenging machine learning benchmarks. Our method raises the potential that standard machine learning algorithms could in principle be directly implemented in neural circuitry.
  arXiv  Detail & Related papers  (2020-06-07T15:35:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.