LLM-KT: A Versatile Framework for Knowledge Transfer from Large Language Models to Collaborative Filtering

• URL: http://arxiv.org/abs/2411.00556v1
• Date: Fri, 01 Nov 2024 13:09:30 GMT
• Title: LLM-KT: A Versatile Framework for Knowledge Transfer from Large Language Models to Collaborative Filtering
• Authors: Nikita Severin, Aleksei Ziablitsev, Yulia Savelyeva, Valeriy Tashchilin, Ivan Bulychev, Mikhail Yushkov, Artem Kushneruk, Amaliya Zaryvnykh, Dmitrii Kiselev, Andrey Savchenko, Ilya Makarov,
• Abstract summary: We present a flexible framework designed to enhance collaborative filtering (CF) models by seamlessly integrating LLM-generated features. Our framework injects these features into an intermediate layer of any CF model, allowing the model to reconstruct and leverage the embeddings internally. Our framework is built for easy integration and modification, providing researchers and developers with a powerful tool for extending CF model capabilities.
• Score: 0.07793154724386657
• License:
• Abstract: We present LLM-KT, a flexible framework designed to enhance collaborative filtering (CF) models by seamlessly integrating LLM (Large Language Model)-generated features. Unlike existing methods that rely on passing LLM-generated features as direct inputs, our framework injects these features into an intermediate layer of any CF model, allowing the model to reconstruct and leverage the embeddings internally. This model-agnostic approach works with a wide range of CF models without requiring architectural changes, making it adaptable to various recommendation scenarios. Our framework is built for easy integration and modification, providing researchers and developers with a powerful tool for extending CF model capabilities through efficient knowledge transfer. We demonstrate its effectiveness through experiments on the MovieLens and Amazon datasets, where it consistently improves baseline CF models. Experimental studies showed that LLM-KT is competitive with the state-of-the-art methods in context-aware settings but can be applied to a broader range of CF models than current approaches.

Related papers

• Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures [15.645254436094055]
  Federated Learning (FL) enables collaborative fine-tuning of Large Language Models without data sharing. We propose FedAMoLE, a lightweight personalized FL framework that enables data-driven heterogeneous model architectures. Experiments show that FedAMoLE improves accuracy by an average of 5.14% compared to existing approaches.
  arXiv  Detail & Related papers  (2024-11-28T13:20:38Z)
• Enhancing Financial Domain Adaptation of Language Models via Model Augmentation [2.9960693856871545]
  This study demonstrates the effectiveness of Composition to Augment Language Models (CALM) in adapting to the financial domain. We developed a CALM to enhance the financial performance of an LLM with strong response capabilities.
  arXiv  Detail & Related papers  (2024-11-14T07:28:09Z)
• Model-GLUE: Democratized LLM Scaling for A Large Model Zoo in the Wild [84.57103623507082]
  This paper introduces Model-GLUE, a holistic Large Language Models scaling guideline. We benchmark existing scaling techniques, especially selective merging, and variants of mixture. We then formulate an optimal strategy for the selection and aggregation of a heterogeneous model zoo. Our methodology involves the clustering of mergeable models and optimal merging strategy selection, and the integration of clusters.
  arXiv  Detail & Related papers  (2024-10-07T15:55:55Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models [42.891427362223176]
  Large language models (LLMs) based on decoder-only transformers have demonstrated superior text understanding capabilities. We propose a novel framework to fully harness the capabilities of LLMs. We further design an LLM-Infused Diffusion Transformer (LI-DiT) based on the framework.
  arXiv  Detail & Related papers  (2024-06-17T17:59:43Z)
• When Life gives you LLMs, make LLM-ADE: Large Language Models with Adaptive Data Engineering [0.0]
  LLM-ADE is a methodology for continued pre-training of large language models. It addresses the challenges of catastrophic forgetting and double descent. It enhances model adaptability to new data while preserving previously acquired knowledge.
  arXiv  Detail & Related papers  (2024-04-19T17:43:26Z)
• Model Composition for Multimodal Large Language Models [71.5729418523411]
  We propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model. Our basic implementation, NaiveMC, demonstrates the effectiveness of this paradigm by reusing modality encoders and merging LLM parameters.
  arXiv  Detail & Related papers  (2024-02-20T06:38:10Z)
• Knowledge Fusion of Large Language Models [73.28202188100646]
  This paper introduces the notion of knowledge fusion for large language models (LLMs) We externalize their collective knowledge and unique strengths, thereby elevating the capabilities of the target model beyond those of any individual source LLM. Our findings confirm that the fusion of LLMs can improve the performance of the target model across a range of capabilities such as reasoning, commonsense, and code generation.
  arXiv  Detail & Related papers  (2024-01-19T05:02:46Z)
• CoLLiE: Collaborative Training of Large Language Models in an Efficient Way [59.09824823710863]
  CoLLiE is an efficient library that facilitates collaborative training of large language models. With its modular design and comprehensive functionality, CoLLiE offers a balanced blend of efficiency, ease of use, and customization.
  arXiv  Detail & Related papers  (2023-12-01T08:02:16Z)
• Efficient Data-specific Model Search for Collaborative Filtering [56.60519991956558]
  Collaborative filtering (CF) is a fundamental approach for recommender systems. In this paper, motivated by the recent advances in automated machine learning (AutoML), we propose to design a data-specific CF model. Key here is a new framework that unifies state-of-the-art (SOTA) CF methods and splits them into disjoint stages of input encoding, embedding function, interaction and prediction function.
  arXiv  Detail & Related papers  (2021-06-14T14:30:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.