AlphaFuse: Learn ID Embeddings for Sequential Recommendation in Null Space of Language Embeddings

• URL: http://arxiv.org/abs/2504.19218v2
• Date: Tue, 29 Apr 2025 12:59:17 GMT
• Title: AlphaFuse: Learn ID Embeddings for Sequential Recommendation in Null Space of Language Embeddings
• Authors: Guoqing Hu, An Zhang, Shuo Liu, Zhibo Cai, Xun Yang, Xiang Wang,
• Abstract summary: We introduce AlphaFuse, a language-guided learning strategy that learns ID embeddings within the null space of language embeddings.<n>Specifically, we decompose the semantic space of language embeddings via Singular Value Decomposition (SVD), distinguishing it into a semantic-rich row space and a semantic-sparse null space.<n>AlphaFuse prevents degradation of the semantic space, integrates the retained language embeddings into the final item embeddings, and eliminates the need for auxiliary trainable modules.
• Score: 17.531288777723297
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in sequential recommendation have underscored the potential of Large Language Models (LLMs) for enhancing item embeddings. However, existing approaches face three key limitations: 1) the degradation of the semantic space when high-dimensional language embeddings are mapped to lower-dimensional ID embeddings, 2) the underutilization of language embeddings, and 3) the reliance on additional trainable parameters, such as an adapter, to bridge the gap between the semantic and behavior spaces. In this paper, we introduce AlphaFuse, a simple but effective language-guided learning strategy that addresses these challenges by learning ID embeddings within the null space of language embeddings. Specifically, we decompose the semantic space of language embeddings via Singular Value Decomposition (SVD), distinguishing it into a semantic-rich row space and a semantic-sparse null space. Collaborative signals are then injected into the null space, while preserving the rich semantics of the row space. AlphaFuse prevents degradation of the semantic space, integrates the retained language embeddings into the final item embeddings, and eliminates the need for auxiliary trainable modules, enabling seamless adaptation to any sequential recommendation framework. We validate the effectiveness and flexibility of AlphaFuse through extensive experiments on three benchmark datasets, including cold-start user and long-tail settings, showcasing significant improvements in both discriminative and diffusion-based generative sequential recommenders. Our codes and datasets are available at https://github.com/Hugo-Chinn/AlphaFuse.

Related papers

• Fine-grained Semantics Integration for Large Language Model-based Recommendation [35.75224379727093]
  We propose TS-Rec, which can integrate Token-level Semantics into LLM-based Recommenders.<n>Extensive experiments on two real-world benchmarks demonstrate that TS-Rec consistently outperforms traditional and generative baselines.
  arXiv  Detail & Related papers  (2026-02-26T05:17:24Z)
• DOS: Dual-Flow Orthogonal Semantic IDs for Recommendation in Meituan [8.259886050799922]
  We propose Dual-Flow Orthogonal Semantic IDs (DOS) method for generative recommendation systems.<n>DOS employs a user-item dual flow-framework that leverages collaborative signals to align the Semantic ID codebook space with the generation space.<n>DOS has been successfully deployed in Meituan's mobile application, serving hundreds of millions of users.
  arXiv  Detail & Related papers  (2026-02-04T11:43:42Z)
• Decoding in Latent Spaces for Efficient Inference in LLM-based Recommendation [75.72196852363116]
  Light Latent-space Decoding (L2D) is an effective and efficient latent-space decoding method.<n>L2D is more than 10x faster than language-space decoding while maintaining or enhancing performance.
  arXiv  Detail & Related papers  (2025-09-15T02:30:35Z)
• HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization [50.27950279695363]
  Many pre-trained language models (PLMs) exhibit suboptimal performance on mid- and low-resource languages.<n>A common strategy to address this is to introduce new tokens specific to the target languages, initialize their embeddings, and apply continual pre-training on target-language data.<n>We propose HYPEROFA, a hypernetwork-based approach for more adaptive token embedding.
  arXiv  Detail & Related papers  (2025-04-21T19:40:32Z)
• Language Representations Can be What Recommenders Need: Findings and Potentials [57.90679739598295]
  We show that item representations, when linearly mapped from advanced LM representations, yield superior recommendation performance.<n>This outcome suggests the possible homomorphism between the advanced language representation space and an effective item representation space for recommendation.<n>Our findings highlight the connection between language modeling and behavior modeling, which can inspire both natural language processing and recommender system communities.
  arXiv  Detail & Related papers  (2024-07-07T17:05:24Z)
• GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplane [53.388937705785025]
  3D open-vocabulary scene understanding is crucial for advancing augmented reality and robotic applications. We introduce GOI, a framework that integrates semantic features from 2D vision-language foundation models into 3D Gaussian Splatting (3DGS) Our method treats the feature selection process as a hyperplane division within the feature space, retaining only features that are highly relevant to the query.
  arXiv  Detail & Related papers  (2024-05-27T18:57:18Z)
• Learnable Item Tokenization for Generative Recommendation [78.30417863309061]
  We propose LETTER (a LEarnable Tokenizer for generaTivE Recommendation), which integrates hierarchical semantics, collaborative signals, and code assignment diversity. LETTER incorporates Residual Quantized VAE for semantic regularization, a contrastive alignment loss for collaborative regularization, and a diversity loss to mitigate code assignment bias.
  arXiv  Detail & Related papers  (2024-05-12T15:49:38Z)
• SPACE-IDEAS: A Dataset for Salient Information Detection in Space Innovation [0.3017070810884304]
  We introduce SPACE-IDEAS, a dataset for salient information detection from innovation ideas related to the Space domain. The text in SPACE-IDEAS varies greatly and includes informal, technical, academic and business-oriented writing styles. In addition to a manually annotated dataset we release an extended version that is annotated using a large generative language model.
  arXiv  Detail & Related papers  (2024-03-25T17:04:02Z)
• Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations [38.56175462620892]
  Large pretrained multilingual language models (ML-LMs) have shown remarkable capabilities of zero-shot cross-lingual transfer. We present a novel view of projecting away language-specific factors from a multilingual embedding space. We show that applying our method consistently leads to improvements over commonly used ML-LMs.
  arXiv  Detail & Related papers  (2024-01-11T09:54:11Z)
• OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining [49.213120730582354]
  Instead of pretraining multilingual language models from scratch, a more efficient method is to adapt existing pretrained language models (PLMs) to new languages via vocabulary extension and continued pretraining. We propose a novel framework: $textbfO$ne $textbfF$or $textbfA$ll, which wisely initializes the embeddings of unseen subwords and thus can adapt a PLM to multiple languages efficiently and effectively.
  arXiv  Detail & Related papers  (2023-11-15T10:40:45Z)
• Language Models As Semantic Indexers [78.83425357657026]
  We introduce LMIndexer, a self-supervised framework to learn semantic IDs with a generative language model. We show the high quality of the learned IDs and demonstrate their effectiveness on three tasks including recommendation, product search, and document retrieval.
  arXiv  Detail & Related papers  (2023-10-11T18:56:15Z)
• Towards Robust and Semantically Organised Latent Representations for Unsupervised Text Style Transfer [6.467090475885798]
  We introduce EPAAEs (versading Perturbed Adrial AutoEncoders) which completes this perturbation model. We empirically show that this (a) produces a better organised latent space that clusters stylistically similar sentences together. We also extend the text style transfer tasks to NLI datasets and show that these more complex definitions of style are learned best by EPAAE.
  arXiv  Detail & Related papers  (2022-05-04T20:04:24Z)
• Integrating Language Guidance into Vision-based Deep Metric Learning [78.18860829585182]
  We propose to learn metric spaces which encode semantic similarities as embedding space. These spaces should be transferable to classes beyond those seen during training. This causes learned embedding spaces to encode incomplete semantic context and misrepresent the semantic relation between classes.
  arXiv  Detail & Related papers  (2022-03-16T11:06:50Z)
• Inducing Language-Agnostic Multilingual Representations [61.97381112847459]
  Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world. We examine three approaches for this: (i) re-aligning the vector spaces of target languages to a pivot source language; (ii) removing language-specific means and variances, which yields better discriminativeness of embeddings as a by-product; and (iii) increasing input similarity across languages by removing morphological contractions and sentence reordering.
  arXiv  Detail & Related papers  (2020-08-20T17:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.