InLINE: Inner-Layer Information Exchange for Multi-task Learning on Heterogeneous Graphs

• URL: http://arxiv.org/abs/2410.22089v1
• Date: Tue, 29 Oct 2024 14:46:49 GMT
• Title: InLINE: Inner-Layer Information Exchange for Multi-task Learning on Heterogeneous Graphs
• Authors: Xinyue Feng, Jinquan Hang, Yuequn Zhang, Haotian Wang, Desheng Zhang, Guang Wang,
• Abstract summary: Heterogeneous graph is an important structure for modeling complex data in real-world scenarios. We propose the Inner-Layer Information Exchange model that facilitate fine-grained information exchanges within each graph layer. Our model effectively alleviates the significant performance drop on specific tasks caused by negative transfer.
• Score: 13.204120407041195
• License:
• Abstract: Heterogeneous graph is an important structure for modeling complex relational data in real-world scenarios and usually involves various node prediction tasks within a single graph. Training these tasks separately may neglect beneficial information sharing, hence a preferred way is to learn several tasks in a same model by Multi-Task Learning (MTL). However, MTL introduces the issue of negative transfer, where the training of different tasks interferes with each other as they may focus on different information from the data, resulting in suboptimal performance. To solve the issue, existing MTL methods use separate backbones for each task, then selectively exchange beneficial features through interactions among the output embeddings from each layer of different backbones, which we refer to as outer-layer exchange. However, the negative transfer in heterogeneous graphs arises not simply from the varying importance of an individual node feature across tasks, but also from the varying importance of inter-relation between two nodes across tasks. These inter-relations are entangled in the output embedding, making it difficult for existing methods to discriminate beneficial information from the embedding. To address this challenge, we propose the Inner-Layer Information Exchange (InLINE) model that facilitate fine-grained information exchanges within each graph layer rather than through output embeddings. Specifically, InLINE consists of (1) Structure Disentangled Experts for layer-wise structure disentanglement, (2) Structure Disentangled Gates for assigning disentangled information to different tasks. Evaluations on two public datasets and a large industry dataset show that our model effectively alleviates the significant performance drop on specific tasks caused by negative transfer, improving Macro F1 by 6.3% on DBLP dataset and AUC by 3.6% on the industry dataset compared to SoA methods.

Related papers

• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Factorized Contrastive Learning: Going Beyond Multi-view Redundancy [116.25342513407173]
  This paper proposes FactorCL, a new multimodal representation learning method to go beyond multi-view redundancy. On large-scale real-world datasets, FactorCL captures both shared and unique information and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2023-06-08T15:17:04Z)
• Curriculum Modeling the Dependence among Targets with Multi-task Learning for Financial Marketing [26.80709680959278]
  We propose a prior information merged model (textbfPIMM) for multiple sequential dependence task learning. The PIM randomly selects the true label information or the prior task prediction with a soft sampling strategy to transfer to the downstream task during the training. The offline experimental results on both public and product datasets verify that PIMM outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-04-25T07:55:16Z)
• Self-Supervised Graph Neural Network for Multi-Source Domain Adaptation [51.21190751266442]
  Domain adaptation (DA) tries to tackle the scenarios when the test data does not fully follow the same distribution of the training data. By learning from large-scale unlabeled samples, self-supervised learning has now become a new trend in deep learning. We propose a novel textbfSelf-textbfSupervised textbfGraph Neural Network (SSG) to enable more effective inter-task information exchange and knowledge sharing.
  arXiv  Detail & Related papers  (2022-04-08T03:37:56Z)
• MGA-VQA: Multi-Granularity Alignment for Visual Question Answering [75.55108621064726]
  Learning to answer visual questions is a challenging task since the multi-modal inputs are within two feature spaces. We propose Multi-Granularity Alignment architecture for Visual Question Answering task (MGA-VQA) Our model splits alignment into different levels to achieve learning better correlations without needing additional data and annotations.
  arXiv  Detail & Related papers  (2022-01-25T22:30:54Z)
• Semi-supervised Multi-task Learning for Semantics and Depth [88.77716991603252]
  Multi-Task Learning (MTL) aims to enhance the model generalization by sharing representations between related tasks for better performance. We propose the Semi-supervised Multi-Task Learning (MTL) method to leverage the available supervisory signals from different datasets. We present a domain-aware discriminator structure with various alignment formulations to mitigate the domain discrepancy issue among datasets.
  arXiv  Detail & Related papers  (2021-10-14T07:43:39Z)
• Multi-Relational Graph based Heterogeneous Multi-Task Learning in Community Question Answering [28.91133131424694]
  We develop a multi-relational graph based Multi-Task Learning model called Heterogeneous Multi-Task Graph Isomorphism Network (HMTGIN) In each training forward pass, HMTGIN embeds the input CQA forum graph by an extension of Graph Isomorphism Network and skip connections. In the evaluation, the embeddings are shared among different task-specific output layers to make corresponding predictions.
  arXiv  Detail & Related papers  (2021-09-04T03:19:20Z)
• Understanding and Improving Information Transfer in Multi-Task Learning [14.43111978531182]
  We study an architecture with a shared module for all tasks and a separate output module for each task. We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer. Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning.
  arXiv  Detail & Related papers  (2020-05-02T23:43:52Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.