QV May Be Enough: Toward the Essence of Attention in LLMs

• URL: http://arxiv.org/abs/2603.15665v1
• Date: Wed, 11 Mar 2026 14:08:53 GMT
• Title: QV May Be Enough: Toward the Essence of Attention in LLMs
• Abstract summary: We introduce the QV paradigm and provide empirical evidence for its validity.<n>We propose the QV-Ka optimization scheme, which is substantiated through experimental validation.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Starting from first principles and a linguistic perspective centered on part-of-speech (POS) and syntactic analysis, this paper explores and derives the underlying essence of the Query-Key-Value (QKV) mechanism within the Transformer architecture. Based on this theoretical foundation, we provide a unified explanatory framework for the efficacy of contemporary architectures, including MQA, GQA, and MLA, while identifying their inherent trade-offs and potential optimization trajectories. We introduce the QV paradigm and provide empirical evidence for its validity. Building upon this, we propose the QV-Ka optimization scheme, which is further substantiated through experimental validation. The interpretable theoretical analysis of the QKV mechanism presented in this work establishes a robust foundation for the future evolution of large language model architectures.

Related papers

• Disentanglement-Based Equivariant Learning for Compositional VQA [51.109034846842405]
  We introduce a novel Disentanglement EquivAriant Learning framework for compositional VQA.<n>We employ causality-inspired interventions to disentangle concepts derived from visual and textual inputs within a re-encoding framework.
  arXiv  Detail & Related papers  (2026-06-01T12:28:37Z)
• Deep Unfolding: Recent Developments, Theory, and Design Guidelines [99.63555420898554]
  This article provides a tutorial-style overview of deep unfolding, a framework that transforms optimization algorithms into structured, trainable ML architectures.<n>We review the foundations of optimization for inference and for learning, introduce four representative design paradigms for deep unfolding, and discuss the distinctive training schemes that arise from their iterative nature.
  arXiv  Detail & Related papers  (2025-12-03T13:16:35Z)
• Explainable Chain-of-Thought Reasoning: An Empirical Analysis on State-Aware Reasoning Dynamics [69.00587226225232]
  We introduce a state-aware transition framework that abstracts CoT trajectories into structured latent dynamics.<n>To characterize the global structure of reasoning, we model their progression as a Markov chain.<n>This abstraction supports a range of analyses, including semantic role identification, temporal pattern visualization, and consistency evaluation.
  arXiv  Detail & Related papers  (2025-08-29T18:53:31Z)
• Foundations of Relational Quantum Field Theory I: Scalars [0.0]
  We develop foundations for a relational approach to quantum field theory (RQFT) based on the operational quantum reference frames (QRFs) framework considered in a relativistic setting.<n>We focus on scalar fields in Minkowski spacetime and discuss the emergence of relational local (bounded) observables and (pointwise) fields.<n>We show how the properties of algebras generated by relational local observables extend the core axioms of Algebraic QFT.
  arXiv  Detail & Related papers  (2025-07-29T09:01:38Z)
• Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey [69.45421620616486]
  This work presents the first structured taxonomy and analysis of discrete tokenization methods designed for large language models (LLMs)<n>We categorize 8 representative VQ variants that span classical and modern paradigms and analyze their algorithmic principles, training dynamics, and integration challenges with LLM pipelines.<n>We identify key challenges including codebook collapse, unstable gradient estimation, and modality-specific encoding constraints.
  arXiv  Detail & Related papers  (2025-07-21T10:52:14Z)
• Towards Analyzing and Understanding the Limitations of VAPO: A Theoretical Perspective [6.963986923957048]
  VAPO is a framework for reinforcement learning for large language models.<n>It addresses challenges such as value model bias, heterogeneous sequence lengths, and sparse reward signals.<n>This paper explores VAPO from a theoretical perspective, highlighting areas where its assumptions might be challenged.
  arXiv  Detail & Related papers  (2025-05-23T15:03:41Z)
• The Foundations of Tokenization: Statistical and Computational Concerns [51.370165245628975]
  Tokenization is a critical step in the NLP pipeline.<n>Despite its recognized importance as a standard representation method in NLP, the theoretical underpinnings of tokenization are not yet fully understood.<n>The present paper contributes to addressing this theoretical gap by proposing a unified formal framework for representing and analyzing tokenizer models.
  arXiv  Detail & Related papers  (2024-07-16T11:12:28Z)
• Rethinking Word-Level Auto-Completion in Computer-Aided Translation [76.34184928621477]
  Word-Level Auto-Completion (WLAC) plays a crucial role in Computer-Assisted Translation. It aims at providing word-level auto-completion suggestions for human translators. We introduce a measurable criterion to answer this question and discover that existing WLAC models often fail to meet this criterion. We propose an effective approach to enhance WLAC performance by promoting adherence to the criterion.
  arXiv  Detail & Related papers  (2023-10-23T03:11:46Z)
• Empirical Translation Process Research: Past and Possible Future Perspectives [0.0]
  This article traces the evolution of empirical TPR within the CRITT TPR-DB tradition. It proposes the Free Energy Principle (FEP) and Active Inference (AIF) as a framework for modeling deeply embedded translation processes.
  arXiv  Detail & Related papers  (2023-08-02T18:22:49Z)
• Benchmarking Machine Reading Comprehension: A Psychological Perspective [45.85089157315507]
  Machine reading comprehension (MRC) has received considerable attention as a benchmark for natural language understanding. The conventional task design of MRC lacks explainability beyond the model interpretation. This paper provides a theoretical basis for the design of MRC datasets based on psychology as well as psychometrics.
  arXiv  Detail & Related papers  (2020-04-04T11:45:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.