Elementwise Language Representation

• URL: http://arxiv.org/abs/2302.13475v1
• Date: Mon, 27 Feb 2023 02:15:56 GMT
• Title: Elementwise Language Representation
• Abstract summary: We propose a new technique for computational language representation called elementwise embedding. We show that elementwise embedding replaces the $w$-dimensional embedding table of a transformer model with $256$ $c$-dimensional elements. BERT trained with elementwise embedding outperforms its subword equivalence in multilabel patent document classification.
• Score: 6.233117407988574
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a new technique for computational language representation called elementwise embedding, in which a material (semantic unit) is abstracted into a horizontal concatenation of lower-dimensional element (character) embeddings. While elements are always characters, materials are arbitrary levels of semantic units so it generalizes to any type of tokenization. To focus only on the important letters, the $n^{th}$ spellings of each semantic unit are aligned in $n^{th}$ attention heads, then concatenated back into original forms creating unique embedding representations; they are jointly projected thereby determining own contextual importance. Technically, this framework is achieved by passing a sequence of materials, each consists of $v$ elements, to a transformer having $h=v$ attention heads. As a pure embedding technique, elementwise embedding replaces the $w$-dimensional embedding table of a transformer model with $256$ $c$-dimensional elements (each corresponding to one of UTF-8 bytes) where $c=w/v$. Using this novel approach, we show that the standard transformer architecture can be reused for all levels of language representations and be able to process much longer sequences at the same time-complexity without "any" architectural modification and additional overhead. BERT trained with elementwise embedding outperforms its subword equivalence (original implementation) in multilabel patent document classification exhibiting superior robustness to domain-specificity and data imbalance, despite using $0.005\%$ of embedding parameters. Experiments demonstrate the generalizability of the proposed method by successfully transferring these enhancements to differently architected transformers CANINE and ALBERT.

Related papers

• All You Need Is Non-Commutative Words [0.0]
  We represent lexical tokens as unitary matrices and encode each sentence as their ordered product.<n>The same algebra yields several capabilities, including antisymmetric self-attention with no query, key, or value projections.<n>We achieve higher accuracy on IMDB and comparable performance on AG News.
  arXiv  Detail & Related papers  (2026-08-29T14:56:57Z)
• Controlling Embedding Spaces with Text-Conditioned Transformations [54.94594915955756]
  We propose a text-conditioned transformation of visual embeddings that makes attributes explicitly accessible.<n> Conditioning on text enables it to learn many attributes simultaneously, accessing them at inference time through an intuitive interface.
  arXiv  Detail & Related papers  (2026-07-24T21:14:29Z)
• Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space [56.37266873329401]
  Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density.<n>We propose $textbfDynamic Large Concept Models (DLCM)$, a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts from tokens to a compressed concept space where reasoning is more efficient.
  arXiv  Detail & Related papers  (2025-12-31T04:19:33Z)
• Transparent Semantic Spaces: A Categorical Approach to Explainable Word Embeddings [0.7200655637873443]
  The paper introduces a novel framework based on category theory to enhance the explainability of artificial intelligence systems.<n>Key topics include the construction of categories $ L_T $ and $ P_T $, providing schematic representations of the semantics of a text.
  arXiv  Detail & Related papers  (2025-08-28T12:19:34Z)
• Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations [0.0]
  We construct Transformer models where the embedding layer is entirely frozen.<n>Our method is compatible with any tokenizer, including a novel Unicode-centric tokenizer.<n>Despite the absence of trainable, semantically embeddings, our models converge, generate coherent text, and, critically, outperform architecturally identical models with trainable embeddings.
  arXiv  Detail & Related papers  (2025-07-07T11:17:32Z)
• Inceptive Transformers: Enhancing Contextual Representations through Multi-Scale Feature Learning Across Domains and Languages [2.98683507969764]
  We introduce textitInceptive Transformer, a modular and lightweight architecture that enriches transformer-based token representations.<n>Our model is designed to balance local and global dependencies by dynamically weighting tokens based on their relevance to a particular task.
  arXiv  Detail & Related papers  (2025-05-26T19:59:22Z)
• PaTH Attention: Position Encoding via Accumulating Householder Transformations [56.32365080761523]
  PaTH is a flexible data-dependent position encoding scheme based on accumulated products of Householder transformations.<n>We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices.
  arXiv  Detail & Related papers  (2025-05-22T08:36:09Z)
• Towards Generalized and Training-Free Text-Guided Semantic Manipulation [123.80467566483038]
  Text-guided semantic manipulation refers to semantically editing an image generated from a source prompt to match a target prompt. We propose a novel $textitGTF$ for text-guided semantic manipulation, which has the following attractive capabilities. Our experiments demonstrate the efficacy of our approach, highlighting its potential to advance the state-of-the-art in semantics manipulation.
  arXiv  Detail & Related papers  (2025-04-24T05:54:56Z)
• Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models [3.382910438968506]
  Tokenization is a fundamental step in natural language processing, breaking text into units that computational models can process. We investigate a hierarchical architecture for autoregressive language modelling that combines character-level and word-level processing. We demonstrate, at scales up to 7 billion parameters, that hierarchical transformers match the downstream task performance of subword-tokenizer-based models.
  arXiv  Detail & Related papers  (2025-01-17T17:51:53Z)
• Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence [6.991281327290525]
  We propose a novel approach for learning interchangeable tokens in language models. Our method is designed to address alpha-equivalence, the principle that renaming bound variables in a syntactic expression preserves semantics.
  arXiv  Detail & Related papers  (2024-10-22T16:34:36Z)
• Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation [59.37587762543934]
  This paper studies the problem of weakly open-vocabulary semantic segmentation (WOVSS) Existing methods suffer from a granularity inconsistency regarding the usage of group tokens. We propose the prototypical guidance network (PGSeg) that incorporates multi-modal regularization.
  arXiv  Detail & Related papers  (2023-10-29T13:18:00Z)
• Contrastive Representation Learning for Cross-Document Coreference Resolution of Events and Entities [0.0]
  We present an approach to entity and event coreference resolution utilizing contrastive representation learning. Our method achieves state-of-the-art results on a number of key metrics on the ECB+ corpus.
  arXiv  Detail & Related papers  (2022-05-23T16:30:20Z)
• Generalized Funnelling: Ensemble Learning and Heterogeneous Document Embeddings for Cross-Lingual Text Classification [78.83284164605473]
  emphFunnelling (Fun) is a recently proposed method for cross-lingual text classification. We describe emphGeneralized Funnelling (gFun) as a generalization of Fun. We show that gFun substantially improves over Fun and over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-09-17T23:33:04Z)
• Learning to Look Inside: Augmenting Token-Based Encoders with Character-Level Information [29.633735942273997]
  XRayEmb is a method for retrofitting existing token-based models with character-level information. We show that incorporating XRayEmb's learned vectors into sequences of pre-trained token embeddings helps performance on both autoregressive and masked pre-trained transformer architectures.
  arXiv  Detail & Related papers  (2021-08-01T08:09:26Z)
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization [50.16128796194463]
  We propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. We introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level.
  arXiv  Detail & Related papers  (2021-06-23T22:24:14Z)
• CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters [14.956626084281638]
  We propose a new variant of BERT that drops the wordpiece system altogether and uses a Character-CNN module instead to represent entire words by consulting their characters. We show that this new model improves the performance of BERT on a variety of medical domain tasks while at the same time producing robust, word-level and open-vocabulary representations.
  arXiv  Detail & Related papers  (2020-10-20T15:58:53Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies [60.285091454321055]
  We design a simple and efficient embedding algorithm that learns a small set of anchor embeddings and a sparse transformation matrix. On text classification, language modeling, and movie recommendation benchmarks, we show that ANT is particularly suitable for large vocabulary sizes.
  arXiv  Detail & Related papers  (2020-03-18T13:07:51Z)
• Supervised Quantile Normalization for Low-rank Matrix Approximation [50.445371939523305]
  We learn the parameters of quantile normalization operators that can operate row-wise on the values of $X$ and/or of its factorization $UV$ to improve the quality of the low-rank representation of $X$ itself. We demonstrate the applicability of these techniques on synthetic and genomics datasets.
  arXiv  Detail & Related papers  (2020-02-08T21:06:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.