Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation

• URL: http://arxiv.org/abs/2604.21536v1
• Date: Thu, 23 Apr 2026 10:59:27 GMT
• Title: Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation
• Abstract summary: Large Language Models (LLMs) present opportunities to enhance user understanding with their reasoning capabilities.<n>We present a novel knowledge distillation method that utilizes textual user profile generated by pre-trained LLMs into sequential recommenders.
• Score: 4.127681400898116
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sequential recommender systems have achieved significant success in modeling temporal user behavior but remain limited in capturing rich user semantics beyond interaction patterns. Large Language Models (LLMs) present opportunities to enhance user understanding with their reasoning capabilities, yet existing integration approaches create prohibitive inference costs in real time. To address these limitations, we present a novel knowledge distillation method that utilizes textual user profile generated by pre-trained LLMs into sequential recommenders without requiring LLM inference at serving time. The resulting approach maintains the inference efficiency of traditional sequential models while requiring neither architectural modifications nor LLM fine-tuning.

Related papers

• Beyond Static LLM Policies: Imitation-Enhanced Reinforcement Learning for Recommendation [23.945049006150555]
  Large language models (LLMs) have become critical tools for enhancing user engagement by delivering personalized content across diverse digital platforms.<n>Direct deployment of LLMs as primary recommendation policies presents notable challenges, including persistent latency issues.<n>This paper proposes a novel offline reinforcement learning framework that leverages imitation learning from LLM-generated trajectories.
  arXiv  Detail & Related papers  (2025-10-15T07:28:29Z)
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models? [65.18157595903124]
  This work investigates iterative approximate evaluation for arbitrary prompts.<n>It introduces Model Predictive Prompt Selection (MoPPS), a Bayesian risk-predictive framework.<n>MoPPS reliably predicts prompt difficulty and accelerates training with significantly reduced rollouts.
  arXiv  Detail & Related papers  (2025-07-07T03:20:52Z)
• Enhancing Temporal Sensitivity of Large Language Model for Recommendation with Counterfactual Tuning [8.798364656768657]
  We propose a framework for underlineRecommendation (CETRec)<n> CETRec is grounded in causal inference principles, which allow it to isolate and measure the specific impact of temporal information on recommendation outcomes.<n>Our code is available at https://anonymous.4open.science/r/CETRec-B9CE/.
  arXiv  Detail & Related papers  (2025-07-03T10:11:35Z)
• Process-based Self-Rewarding Language Models [47.119444722849025]
  Large Language Models have demonstrated outstanding performance across various downstream tasks and have been widely applied in multiple scenarios.<n>Human-annotated preference data is used for training to further improve LLMs' performance, which is constrained by the upper limit of human performance.<n>We propose the Process-based Self-Rewarding pipeline for language models, which introduces long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization.
  arXiv  Detail & Related papers  (2025-03-05T18:58:44Z)
• Enhancing Item Tokenization for Generative Recommendation through Self-Improvement [67.94240423434944]
  Generative recommendation systems are driven by large language models (LLMs)<n>Current item tokenization methods include using text descriptions, numerical strings, or sequences of discrete tokens.<n>We propose a self-improving item tokenization method that allows the LLM to refine its own item tokenizations during training process.
  arXiv  Detail & Related papers  (2024-12-22T21:56:15Z)
• Towards a Unified Paradigm: Integrating Recommendation Systems as a New Language in Large Models [33.02146794292383]
  We introduce a new concept, "Integrating Recommendation Systems as a New Language in Large Models" (RSLLM)<n>RSLLM uses a unique prompting method that combines ID-based item embeddings from conventional recommendation models with textual item features.<n>It treats users' sequential behaviors as a distinct language and aligns the ID embeddings with the LLM's input space using a projector.
  arXiv  Detail & Related papers  (2024-12-22T09:08:46Z)
• Real-Time Personalization for LLM-based Recommendation with Customized In-Context Learning [57.28766250993726]
  This work explores adapting to dynamic user interests without any model updates. Existing Large Language Model (LLM)-based recommenders often lose the in-context learning ability during recommendation tuning. We propose RecICL, which customizes recommendation-specific in-context learning for real-time recommendations.
  arXiv  Detail & Related papers  (2024-10-30T15:48:36Z)
• DELRec: Distilling Sequential Pattern to Enhance LLMs-based Sequential Recommendation [7.914816884185941]
  Sequential recommendation (SR) tasks aim to predict users' next interaction by learning their behavior sequence and capturing the connection between users' past interactions and their changing preferences.<n> Conventional SR models often focus solely on capturing sequential patterns within the training data, neglecting the broader context and semantic information embedded in item titles from external sources.<n>Large language models (LLMs) have recently shown promise in SR tasks due to their advanced understanding capabilities and strong generalization abilities.
  arXiv  Detail & Related papers  (2024-06-17T02:47:09Z)
• Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
  We introduce a novel closed-form formulation for direct preference optimization using multiple reference models. The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models. Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
  arXiv  Detail & Related papers  (2024-05-26T00:29:04Z)
• Improve Temporal Awareness of LLMs for Sequential Recommendation [61.723928508200196]
  Large language models (LLMs) have demonstrated impressive zero-shot abilities in solving a wide range of general-purpose tasks. LLMs fall short in recognizing and utilizing temporal information, rendering poor performance in tasks that require an understanding of sequential data. We propose three prompting strategies to exploit temporal information within historical interactions for LLM-based sequential recommendation.
  arXiv  Detail & Related papers  (2024-05-05T00:21:26Z)
• CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning [59.88924847995279]
  We propose a novel Cross-Modal LLM Fine-Tuning (CALF) framework for MTSF.<n>To reduce the distribution discrepancy, we develop the cross-modal match module.<n>CALF establishes state-of-the-art performance for both long-term and short-term forecasting tasks.
  arXiv  Detail & Related papers  (2024-03-12T04:04:38Z)
• Effective and Efficient Training for Sequential Recommendation using Recency Sampling [91.02268704681124]
  We propose a novel Recency-based Sampling of Sequences training objective. We show that the models enhanced with our method can achieve performances exceeding or very close to stateof-the-art BERT4Rec.
  arXiv  Detail & Related papers  (2022-07-06T13:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.