All You Need Is Non-Commutative Words

• URL: http://arxiv.org/abs/2608.29314v1
• Date: Sat, 29 Aug 2026 14:56:57 GMT
• Title: All You Need Is Non-Commutative Words
• Abstract summary: We represent lexical tokens as unitary matrices and encode each sentence as their ordered product.<n>The same algebra yields several capabilities, including antisymmetric self-attention with no query, key, or value projections.<n>We achieve higher accuracy on IMDB and comparable performance on AG News.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We represent lexical tokens as unitary matrices and encode each sentence as their ordered product. The noncommutativity of matrix product captures word order without positional encodings (PEs). The same algebra yields several capabilities, including antisymmetric self-attention with no query, key, or value projections, and parallel composition of variable-length text chunks at a reduced attention cost. Furthermore, it provides a canonical-coset readout layer that encodes all true unitary degrees of freedom compactly, while supporting continual learning through nested group extensions that enlarge the operator space with each new task preserving prior representations exactly. Across standard text-classification benchmarks, the method matches or exceeds bag-of-words baselines. Achieving higher accuracy on IMDB and comparable performance on AG News. Notably, this is accomplished by replacing the conventional $\sim$30,000-dimensional vocabulary space with a dense, 64-parameter real-valued encoding, highlighting the expressive efficiency of our parameterization.

Related papers

• Rethinking Item Tokenization in Generative Recommenders: From Fixed Atoms to Semantic Subwords [50.621722853036005]
  In generative recommender systems, items are typically tokenized into fixed-length semantic ID sequences for autoregressive next-item prediction.<n>For user-context modeling, this fine-grained representation triggers Intra-item Attention Overload.<n>We propose Semantic Subword Tokenization (SST), which represents historical items as variable-length semantic subwords.
  arXiv  Detail & Related papers  (2026-08-24T02:42:51Z)
• Prefix Parsing is Just Parsing [53.010152712125766]
  Prefix parsing asks whether an input prefix can be extended to a complete string generated by a given grammar.<n>We introduce the prefix grammar transformation, an efficient reduction of prefix parsing to ordinary parsing.<n>We also present a strategy-based on algorithmic differentiation-for computing the next-token weight vector.
  arXiv  Detail & Related papers  (2026-04-23T01:20:40Z)
• Breaking Token Into Concepts: Exploring Extreme Compression in Token Representation Via Compositional Shared Semantics [4.822201556735661]
  We investigate whether tokens can be more effectively represented through a compositional structure that accumulates diverse semantic facets.<n>We propose Aggregate Semantic Grouping (ASG), a novel approach leveraging Product Quantization (PQ)<n>Our findings demonstrate that representing tokens compositionally via ASG achieves extreme compression in embedding parameters.<n>These results validate the principle that tokens can be effectively modeled as combinations of shared semantic building blocks.
  arXiv  Detail & Related papers  (2025-09-22T13:04:48Z)
• Tokens with Meaning: A Hybrid Tokenization Approach for NLP [0.2826977330147589]
  Tokenization plays a pivotal role in natural language processing (NLP)<n>We introduce a hybrid tokenization framework that combines rule-based morphological analysis with statistical subword segmentation.<n>The method uses phono normalization, root-affix, and a novel algorithm that balances morpheme preservation with vocabulary efficiency.
  arXiv  Detail & Related papers  (2025-08-19T22:17:42Z)
• Copy Is All You Need [66.00852205068327]
  We formulate text generation as progressively copying text segments from an existing text collection. Our approach achieves better generation quality according to both automatic and human evaluations. Our approach attains additional performance gains by simply scaling up to larger text collections.
  arXiv  Detail & Related papers  (2023-07-13T05:03:26Z)
• Linear-Time Modeling of Linguistic Structure: An Order-Theoretic Perspective [97.57162770792182]
  Tasks that model the relation between pairs of tokens in a string are a vital part of understanding natural language. We show that these exhaustive comparisons can be avoided, and, moreover, the complexity can be reduced to linear by casting the relation between tokens as a partial order over the string. Our method predicts real numbers for each token in a string in parallel and sorts the tokens accordingly, resulting in total orders of the tokens in the string.
  arXiv  Detail & Related papers  (2023-05-24T11:47:35Z)
• Elementwise Language Representation [6.233117407988574]
  We propose a new technique for computational language representation called elementwise embedding. We show that elementwise embedding replaces the $w$-dimensional embedding table of a transformer model with $256$ $c$-dimensional elements. BERT trained with elementwise embedding outperforms its subword equivalence in multilabel patent document classification.
  arXiv  Detail & Related papers  (2023-02-27T02:15:56Z)
• Composable Text Controls in Latent Space with ODEs [97.12426987887021]
  This paper proposes a new efficient approach for composable text operations in the compact latent space of text. By connecting pretrained LMs to the latent space through efficient adaption, we then decode the sampled vectors into desired text sequences. Experiments show that composing those operators within our approach manages to generate or edit high-quality text.
  arXiv  Detail & Related papers  (2022-08-01T06:51:45Z)
• Rethinking Positional Encoding in Language Pre-training [111.2320727291926]
  We show that in absolute positional encoding, the addition operation applied on positional embeddings and word embeddings brings mixed correlations. We propose a new positional encoding method called textbfTransformer with textbfUntied textPositional textbfEncoding (T)
  arXiv  Detail & Related papers  (2020-06-28T13:11:02Z)
• All Word Embeddings from One Embedding [23.643059189673473]
  In neural network-based models for natural language processing, the largest part of the parameters often consists of word embeddings. In this study, to reduce the total number of parameters, the embeddings for all words are represented by transforming a shared embedding. The proposed method, ALONE, constructs the embedding of a word by modifying the shared embedding with a filter vector, which is word-specific but non-trainable.
  arXiv  Detail & Related papers  (2020-04-25T07:38:08Z)
• Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies [60.285091454321055]
  We design a simple and efficient embedding algorithm that learns a small set of anchor embeddings and a sparse transformation matrix. On text classification, language modeling, and movie recommendation benchmarks, we show that ANT is particularly suitable for large vocabulary sizes.
  arXiv  Detail & Related papers  (2020-03-18T13:07:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.