PDT: Pretrained Dual Transformers for Time-aware Bipartite Graphs

• URL: http://arxiv.org/abs/2306.01913v3
• Date: Mon, 25 Sep 2023 17:31:16 GMT
• Title: PDT: Pretrained Dual Transformers for Time-aware Bipartite Graphs
• Authors: Xin Dai, Yujie Fan, Zhongfang Zhuang, Shubham Jain, Chin-Chia Michael Yeh, Junpeng Wang, Liang Wang, Yan Zheng, Prince Osei Aboagye, Wei Zhang
• Abstract summary: We propose a pre-training method that learns a bi-directional mapping between the spaces of the user-side and the content-side. We evaluate the proposed method for the recommendation task.
• Score: 30.71271242109731
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-training on large models is prevalent and emerging with the ever-growing user-generated content in many machine learning application categories. It has been recognized that learning contextual knowledge from the datasets depicting user-content interaction plays a vital role in downstream tasks. Despite several studies attempting to learn contextual knowledge via pre-training methods, finding an optimal training objective and strategy for this type of task remains a challenging problem. In this work, we contend that there are two distinct aspects of contextual knowledge, namely the user-side and the content-side, for datasets where user-content interaction can be represented as a bipartite graph. To learn contextual knowledge, we propose a pre-training method that learns a bi-directional mapping between the spaces of the user-side and the content-side. We formulate the training goal as a contrastive learning task and propose a dual-Transformer architecture to encode the contextual knowledge. We evaluate the proposed method for the recommendation task. The empirical studies have demonstrated that the proposed method outperformed all the baselines with significant gains.

Related papers

• Improving Content Recommendation: Knowledge Graph-Based Semantic Contrastive Learning for Diversity and Cold-Start Users [5.224122150536595]
  We propose a hybrid multi-task learning approach, training on user-item and item-item interactions. Our approach allows the model to better understand the relationships between entities within the knowledge graph by utilizing semantic information from text.
  arXiv  Detail & Related papers  (2024-03-27T15:11:00Z)
• Generative Multi-Modal Knowledge Retrieval with Large Language Models [75.70313858231833]
  We propose an innovative end-to-end generative framework for multi-modal knowledge retrieval. Our framework takes advantage of the fact that large language models (LLMs) can effectively serve as virtual knowledge bases. We demonstrate significant improvements ranging from 3.0% to 14.6% across all evaluation metrics when compared to strong baselines.
  arXiv  Detail & Related papers  (2024-01-16T08:44:29Z)
• Compositional Learning in Transformer-Based Human-Object Interaction Detection [6.630793383852106]
  Long-tailed distribution of labeled instances is a primary challenge in HOI detection. Inspired by the nature of HOI triplets, some existing approaches adopt the idea of compositional learning. We creatively propose a transformer-based framework for compositional HOI learning.
  arXiv  Detail & Related papers  (2023-08-11T06:41:20Z)
• Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding [52.723297744257536]
  Pre-trained language models (LMs) have shown effectiveness in scientific literature understanding tasks. We propose a multi-task contrastive learning framework, SciMult, to facilitate common knowledge sharing across different literature understanding tasks.
  arXiv  Detail & Related papers  (2023-05-23T16:47:22Z)
• Self-Supervised Graph Neural Network for Multi-Source Domain Adaptation [51.21190751266442]
  Domain adaptation (DA) tries to tackle the scenarios when the test data does not fully follow the same distribution of the training data. By learning from large-scale unlabeled samples, self-supervised learning has now become a new trend in deep learning. We propose a novel textbfSelf-textbfSupervised textbfGraph Neural Network (SSG) to enable more effective inter-task information exchange and knowledge sharing.
  arXiv  Detail & Related papers  (2022-04-08T03:37:56Z)
• Leveraging Visual Knowledge in Language Tasks: An Empirical Study on Intermediate Pre-training for Cross-modal Knowledge Transfer [61.34424171458634]
  We study whether integrating visual knowledge into a language model can fill the gap. Our experiments show that visual knowledge transfer can improve performance in both low-resource and fully supervised settings.
  arXiv  Detail & Related papers  (2022-03-14T22:02:40Z)
• Cross-Task Knowledge Distillation in Multi-Task Recommendation [41.62428191434233]
  Multi-task learning has been widely used in real-world recommenders to predict different types of user feedback. We propose a Cross-Task Knowledge Distillation framework in recommendation, which consists of three procedures.
  arXiv  Detail & Related papers  (2022-02-20T16:15:19Z)
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge [59.87823082513752]
  This paper investigates the injection of knowledge from general-purpose knowledge bases (KBs) into vision-and-language transformers. We empirically study the relevance of various KBs to multiple tasks and benchmarks. The technique is model-agnostic and can expand the applicability of any vision-and-language transformer with minimal computational overhead.
  arXiv  Detail & Related papers  (2021-01-15T08:37:55Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.