Enhancing Hyperspace Analogue to Language (HAL) Representations via Attention-Based Pooling for Text Classification

• URL: http://arxiv.org/abs/2603.20149v1
• Date: Fri, 20 Mar 2026 17:25:09 GMT
• Title: Enhancing Hyperspace Analogue to Language (HAL) Representations via Attention-Based Pooling for Text Classification
• Abstract summary: We develop a learnable, temperature-scaled additive attention mechanism into the HAL representation pipeline.<n>We evaluate the proposed architecture on the IMDB sentiment analysis dataset.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Hyperspace Analogue to Language (HAL) model relies on global word co-occurrence matrices to construct distributional semantic representations. While these representations capture lexical relationships effectively, aggregating them into sentence-level embeddings via standard mean pooling often results in information loss. Mean pooling assigns equal weight to all tokens, thereby diluting the impact of contextually salient words with uninformative structural tokens. In this paper, we address this limitation by integrating a learnable, temperature-scaled additive attention mechanism into the HAL representation pipeline. To mitigate the sparsity and high dimensionality of the raw co-occurrence matrices, we apply Truncated Singular Value Decomposition (SVD) to project the vectors into a dense latent space prior to the attention layer. We evaluate the proposed architecture on the IMDB sentiment analysis dataset. Empirical results demonstrate that the attention-based pooling approach achieves a test accuracy of 82.38%, yielding an absolute improvement of 6.74 percentage points over the traditional mean pooling baseline (75.64%). Furthermore, qualitative analysis of the attention weights indicates that the mechanism successfully suppresses stop-words and selectively attends to sentiment-bearing tokens, improving both classification performance and model interpretability.

Related papers

• Disentangling Semantic Attention from Structural Bias in the Attention Manifold [61.68044165974505]
  Multimodal Large Language Models (MLLMs) exhibit disproportionate attention toward certain semantically uninformative visual tokens.<n>We introduce Saliency-guided Purification and Adaptive Redistribution (SPAR), a training-free, plug-and-play intervention.
  arXiv  Detail & Related papers  (2026-07-27T05:30:19Z)
• Towards Explainability of SLMs by investigating Token Level Activation [2.7590644934977764]
  This work introduces a framework for quantifying token-level representational importance using hidden-state activation strengths at Layer 8 of BERT.<n>The proposed Activation Flow Network (AFN) framework computes Token Activation Strength using the L2 norm of Layer-8 hidden representations.<n>The study further introduces a threshold-based activation bucket formulation that partitions tokens into HIGH-activation and LOW-activation groups.
  arXiv  Detail & Related papers  (2026-05-21T12:09:12Z)
• Interpretable Coreference Resolution Evaluation Using Explicit Semantics [35.71215348826291]
  Coreference resolution is evaluated using aggregate statistical metrics such as CoNLL-F1.<n>We introduce a semantically-enhanced evaluation framework for coreference resolution.<n>We show that our framework uncovers systematic weaknesses that remain obscured by aggregate metrics.
  arXiv  Detail & Related papers  (2026-05-11T14:20:16Z)
• SERSEM: Selective Entropy-Weighted Scoring for Membership Inference in Code Language Models [1.327416973220814]
  We propose SERSEM, a novel white-box attack framework that suppresses uninformative syntactical boilerplate to amplify memorization signals.<n>Our findings demonstrate that focusing on human-centric coding anomalies provides a significantly more robust indicator of verbatim memorization than sequence-level probability averages.
  arXiv  Detail & Related papers  (2026-04-01T17:03:58Z)
• Beyond Attention Magnitude: Leveraging Inter-layer Rank Consistency for Efficient Vision-Language-Action Models [75.27089289058613]
  We introduce textbfTIES (textbfTau-guided textbfInter-layer textbfEfficient textbfSelection), a dynamic framework guided by inter-layer token ranking consistency.<n>On the CogACT + SIMPLER benchmark, TIES improves average success rates by 6% while reducing token usage by 78%, and demonstrate strong generalization across diverse decoders and benchmarks.
  arXiv  Detail & Related papers  (2026-03-26T02:13:03Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)
• Bridging the Semantic Gap for Categorical Data Clustering via Large Language Models [64.58262227709842]
  ARISE (Attention-weighted Representation with Integrated Semantic Embeddings) is presented.<n>It builds semantic-aware representations that complement the metric space of categorical data for accurate clustering.<n>Experiments on eight benchmark datasets demonstrate consistent improvements over seven representative counterparts.
  arXiv  Detail & Related papers  (2026-01-03T11:37:46Z)
• Framework for Machine Evaluation of Reasoning Completeness in Large Language Models For Classification Tasks [0.0]
  This paper introduces RACE-Reasoning Alignment for Completeness of Explanations.<n>We analyze four widely used text classification datasets-WIKI ONTOLOGY, AG NEWS, IMDB, and GOEMOTIONS.<n>We show that correct predictions exhibit higher coverage of supporting features, while incorrect predictions are associated with elevated coverage of contradicting features.
  arXiv  Detail & Related papers  (2025-10-23T20:22:22Z)
• Beyond Attention: Learning Spatio-Temporal Dynamics with Emergent Interpretable Topologies [0.20482269513546453]
  Graph Attention Networks (GATs) are popular for modeling spatial dependencies, but they rely on predefined adjacency structures and dynamic attention scores.<n>We propose InterGAT, a simplified alternative that replaces masked attention with a fully learnable, symmetric node interaction matrix.<n>Our framework, InterGAT-GRU, outperforms the baseline GAT-GRU in forecasting accuracy, achieving at least a 21% improvement on the SZ-Taxi dataset and a 6% improvement on the Los-Loop dataset.
  arXiv  Detail & Related papers  (2025-06-01T01:27:32Z)
• Balancing Complexity and Informativeness in LLM-Based Clustering: Finding the Goldilocks Zone [0.0]
  This paper investigates the optimal number of clusters by quantifying the trade-off between informativeness and cognitive simplicity.<n>We use large language models (LLMs) to generate cluster names and evaluate their effectiveness.<n>We identify an optimal range of 16-22 clusters, paralleling linguistic efficiency in lexical categorization.
  arXiv  Detail & Related papers  (2025-04-06T01:16:22Z)
• Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability [53.51560766150442]
  Critical tokens are elements within reasoning trajectories that significantly influence incorrect outcomes.<n>We present a novel framework for identifying these tokens through rollout sampling.<n>We show that identifying and replacing critical tokens significantly improves model accuracy.
  arXiv  Detail & Related papers  (2024-11-29T18:58:22Z)
• Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens [57.37893387775829]
  We introduce a fast and balanced clustering method, named Semantic Equitable Clustering (SEC)<n>SEC clusters tokens based on their global semantic relevance in an efficient, straightforward manner.<n>We propose a versatile vision backbone, SECViT, to serve as a vision language connector.
  arXiv  Detail & Related papers  (2024-05-22T04:49:00Z)
• Markovian Transformers for Informative Language Modeling [1.172865818448696]
  Chain-of-Thought (CoT) reasoning often fails to faithfully reflect a language model's underlying decision process.<n>We introduce a Markovian language model framework that can be understood as a reasoning autoencoder.
  arXiv  Detail & Related papers  (2024-04-29T17:36:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.