Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

• URL: http://arxiv.org/abs/2608.13426v1
• Date: Thu, 13 Aug 2026 16:16:04 GMT
• Title: Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference
• Abstract summary: Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications.<n>We propose Reduced Matrix multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along their contraction dimensions.
• Score: 14.784763071210014
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications. We propose Reduced Matrix Multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along their contraction dimensions, without modifying model weights. Under a simple retention-ratio control, RMM provides a smooth and predictable accuracy-efficiency trade-off. Across language models ranging from 1B to 70B parameters, we find that reduction tolerance depends on the model family, task, component, and retention ratio, although it often improves with model scale. Under moderate reduction, RMM remains robust across the evaluated discriminative, autoregressive generation, and long-context settings. We further show that the same principle extends to multimodal vision-language inference. Mechanistic ablations reveal a structural asymmetry within Transformers: attention-side computations are substantially more reducible than MLP components. Finally, wall-clock benchmarks with custom kernels on an NVIDIA A100 show that these computational savings can translate into practical runtime gains, especially at longer sequence lengths. Together, these results position RMM as a scalable direction for input-adaptive inference-time optimization.

Related papers

• Structured Adaptive Tensor Prediction for Streaming Data [12.618068868826512]
  Matrix-valued time series arise in a wide range of applications, such as adaptability of medical imaging data.<n>Existing methods are mainly designed for static settings and lack to streaming and time-varying environments.<n>We develop an adaptive regression framework that includes Matrix-on-Matrix (MoM) and gradient-on-Matrix (ToM)<n>We show that ToM achieves lower steady-state error and stronger denoising capability than MoM.
  arXiv  Detail & Related papers  (2026-06-08T19:09:58Z)
• When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet [10.055354775930487]
  Matrix inversion in chunk-wise parallel linear attention is a major bottleneck for long-context modeling.<n>We propose a fast, Matrix multiplication (MatMul)-based algorithm tailored for strictly lower-triangular matrices.<n> Experiments on Qwen3.5-family models demonstrate up to 5$times$ kernel-level speedup and a 20% reduction in decode-layer overhead.
  arXiv  Detail & Related papers  (2026-06-04T11:29:05Z)
• POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation [57.57816409869894]
  We introduce POET-X, a scalable and memory-efficient variant for training large language models.<n>PoET-X maintains the generalization and stability benefits of POET while achieving substantial improvements in throughput and memory efficiency.
  arXiv  Detail & Related papers  (2026-03-05T18:59:23Z)
• Tuning-Free Structured Sparse Recovery of Multiple Measurement Vectors using Implicit Regularization [13.378211527081582]
  We introduce a tuning-free framework to recover sparse signals in multiple measurement vectors.<n>We show that the optimization dynamics exhibit a "momentum-like" effect, causing the norms of rows in the true support to grow significantly faster than others.
  arXiv  Detail & Related papers  (2025-12-03T02:53:11Z)
• IMSE: Efficient U-Net-based Speech Enhancement using Inception Depthwise Convolution and Amplitude-Aware Linear Attention [2.3959703715401903]
  This paper proposes IMSE, a systematically optimized and ultra-lightweight network.<n>We introduce two core innovations: 1) replacing the MET module with Amplitude-Aware Linear Attention (MALA) and 2) replacing the Deformable Embedding (DE) module with Inception Depthwise Convolution (IDConv)<n>In experiments, IMSE significantly reduces the parameter count by 16.8% (from 0.513M to 0.427M) while achieving competitive performance comparable to the state-of-the-art on the PESQ metric (3.373)
  arXiv  Detail & Related papers  (2025-11-18T14:11:54Z)
• QR-LoRA: Efficient and Disentangled Fine-tuning via QR Decomposition for Customized Generation [52.024845354511555]
  We propose QR-LoRA, a novel fine-tuning framework leveraging QR decomposition for structured parameter updates.<n>Our key insight is that the Q matrix naturally minimizes interference between different visual features.<n>Experiments demonstrate that QR-LoRA achieves superior disentanglement in content-style fusion tasks.
  arXiv  Detail & Related papers  (2025-07-07T01:31:01Z)
• OSoRA: Output-Dimension and Singular-Value Initialized Low-Rank Adaptation [9.048461365342204]
  We present OSoRA, a novel PEFT method for Large Language Models (LLMs)<n>OSoRA substantially reduces computational resource requirements by minimizing the number of trainable parameters during fine-tuning.<n> Comprehensive evaluations across mathematical reasoning, common sense reasoning, and other benchmarks demonstrate that OSoRA achieves comparable or superior performance to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-20T13:34:06Z)
• Reinforced Model Merging [53.84354455400038]
  We present an innovative framework termed Reinforced Model Merging (RMM), which encompasses an environment and agent tailored for merging tasks.<n>By utilizing data subsets during the evaluation process, we addressed the bottleneck in the reward feedback phase, thereby accelerating RMM by up to 100 times.
  arXiv  Detail & Related papers  (2025-03-27T08:52:41Z)
• Efficient Adaptation of Pre-trained Vision Transformer via Householder Transformation [53.88562288388169]
  A common strategy for. Efficient Fine-Tuning (PEFT) of pre-trained Vision Transformers (ViTs) involves adapting the model to downstream tasks. We propose a novel PEFT approach inspired by Singular Value Decomposition (SVD) for representing the adaptation matrix. SVD decomposes a matrix into the product of a left unitary matrix, a diagonal matrix of scaling values, and a right unitary matrix.
  arXiv  Detail & Related papers  (2024-10-30T12:08:30Z)
• LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning [66.85589263870702]
  Our approach uses an iterative algorithm to decompose each pretrained matrix into a high-precision low-rank component and a memory-efficient quantized component. Experiments on finetuning RoBERTa and LLaMA-2 demonstrate that our low-rank plus quantized matrix decomposition approach (LQ-LoRA) outperforms strong QLoRA and GPTQ-LoRA baselines.
  arXiv  Detail & Related papers  (2023-11-20T18:57:41Z)
• Numerical Optimizations for Weighted Low-rank Estimation on Language Model [73.12941276331316]
  Singular value decomposition (SVD) is one of the most popular compression methods that approximates a target matrix with smaller matrices. Standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. We show that our method can perform better than current SOTA methods in neural-based language models.
  arXiv  Detail & Related papers  (2022-11-02T00:58:02Z)
• Bayesian Transformer Language Models for Speech Recognition [59.235405107295655]
  State-of-the-art neural language models (LMs) represented by Transformers are highly complex. This paper proposes a full Bayesian learning framework for Transformer LM estimation.
  arXiv  Detail & Related papers  (2021-02-09T10:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.