HyperBERT: Mixing Hypergraph-Aware Layers with Language Models for Node Classification on Text-Attributed Hypergraphs

• URL: http://arxiv.org/abs/2402.07309v3
• Date: Sat, 25 May 2024 12:10:38 GMT
• Title: HyperBERT: Mixing Hypergraph-Aware Layers with Language Models for Node Classification on Text-Attributed Hypergraphs
• Authors: Adrián Bazaga, Pietro Liò, Gos Micklem,
• Abstract summary: We propose a new architecture, HyperBERT, which simultaneously models hypergraph relational structure and text-attributed hypergraphs. Results show that HyperBERT achieves a new state-of-the-art on five challenging text-attributed hypergraph node classification benchmarks.
• Score: 16.07396492960869
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hypergraphs are characterized by complex topological structure, representing higher-order interactions among multiple entities through hyperedges. Lately, hypergraph-based deep learning methods to learn informative data representations for the problem of node classification on text-attributed hypergraphs have garnered increasing research attention. However, existing methods struggle to simultaneously capture the full extent of hypergraph structural information and the rich linguistic attributes inherent in the nodes attributes, which largely hampers their effectiveness and generalizability. To overcome these challenges, we explore ways to further augment a pretrained BERT model with specialized hypergraph-aware layers for the task of node classification. Such layers introduce higher-order structural inductive bias into the language model, thus improving the model's capacity to harness both higher-order context information from the hypergraph structure and semantic information present in text. In this paper, we propose a new architecture, HyperBERT, a mixed text-hypergraph model which simultaneously models hypergraph relational structure while maintaining the high-quality text encoding capabilities of a pre-trained BERT. Notably, HyperBERT presents results that achieve a new state-of-the-art on five challenging text-attributed hypergraph node classification benchmarks.

Related papers

• Hypergraph Transformer for Semi-Supervised Classification [50.92027313775934]
  We propose a novel hypergraph learning framework, HyperGraph Transformer (HyperGT) HyperGT uses a Transformer-based neural network architecture to effectively consider global correlations among all nodes and hyperedges. It achieves comprehensive hypergraph representation learning by effectively incorporating global interactions while preserving local connectivity patterns.
  arXiv  Detail & Related papers  (2023-12-18T17:50:52Z)
• Hypergraph-MLP: Learning on Hypergraphs without Message Passing [41.43504601820411]
  Many hypergraph neural networks leverage message passing over hypergraph structures to enhance node representation learning. We propose an alternative approach where we integrate the information about hypergraph structures into training supervision without explicit message passing. Specifically, we introduce Hypergraph-MLP, a novel learning framework for hypergraph-structured data.
  arXiv  Detail & Related papers  (2023-12-15T13:30:04Z)
• Learning from Heterogeneity: A Dynamic Learning Framework for Hypergraphs [24.68703120825866]
  We propose a hypergraph learning framework named LFH that is capable of dynamic hyperedge construction and attentive embedding update. To evaluate the effectiveness of our proposed framework, we conduct comprehensive experiments on several popular datasets.
  arXiv  Detail & Related papers  (2023-07-07T06:26:44Z)
• From Hypergraph Energy Functions to Hypergraph Neural Networks [94.88564151540459]
  We present an expressive family of parameterized, hypergraph-regularized energy functions. We then demonstrate how minimizers of these energies effectively serve as node embeddings. We draw parallels between the proposed bilevel hypergraph optimization, and existing GNN architectures in common use.
  arXiv  Detail & Related papers  (2023-06-16T04:40:59Z)
• Tensorized Hypergraph Neural Networks [69.65385474777031]
  We propose a novel adjacency-tensor-based textbfTensorized textbfHypergraph textbfNeural textbfNetwork (THNN) THNN is faithful hypergraph modeling framework through high-order outer product feature passing message. Results from experiments on two widely used hypergraph datasets for 3-D visual object classification show the model's promising performance.
  arXiv  Detail & Related papers  (2023-06-05T03:26:06Z)
• Augmentations in Hypergraph Contrastive Learning: Fabricated and Generative [126.0985540285981]
  We apply the contrastive learning approach from images/graphs (we refer to it as HyperGCL) to improve generalizability of hypergraph neural networks. We fabricate two schemes to augment hyperedges with higher-order relations encoded, and adopt three augmentation strategies from graph-structured data. We propose a hypergraph generative model to generate augmented views, and then an end-to-end differentiable pipeline to jointly learn hypergraph augmentations and model parameters.
  arXiv  Detail & Related papers  (2022-10-07T20:12:20Z)
• Equivariant Hypergraph Diffusion Neural Operators [81.32770440890303]
  Hypergraph neural networks (HNNs) using neural networks to encode hypergraphs provide a promising way to model higher-order relations in data. This work proposes a new HNN architecture named ED-HNN, which provably represents any continuous equivariant hypergraph diffusion operators. We evaluate ED-HNN for node classification on nine real-world hypergraph datasets.
  arXiv  Detail & Related papers  (2022-07-14T06:17:00Z)
• Hypergraph Convolutional Networks via Equivalency between Hypergraphs and Undirected Graphs [59.71134113268709]
  We present General Hypergraph Spectral Convolution(GHSC), a general learning framework that can handle EDVW and EIVW hypergraphs. In this paper, we show that the proposed framework can achieve state-of-the-art performance. Experiments from various domains including social network analysis, visual objective classification, protein learning demonstrate that the proposed framework can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-03-31T10:46:47Z)
• Adaptive Neural Message Passing for Inductive Learning on Hypergraphs [21.606287447052757]
  We present HyperMSG, a novel hypergraph learning framework. It adapts to the data and task by learning an attention weight associated with each node's degree centrality. It is robust and outperforms state-of-the-art hypergraph learning methods on a wide range of tasks and datasets.
  arXiv  Detail & Related papers  (2021-09-22T12:24:02Z)
• Learnable Hypergraph Laplacian for Hypergraph Learning [34.28748027233654]
  HyperGraph Convolutional Neural Networks (HGCNNs) have demonstrated their potential in modeling high-order relations preserved in graph structured data. We propose the first learning-based method tailored for constructing adaptive hypergraph structure, termed HypERgrAph Laplacian aDaptor (HERALD) HERALD adaptively optimize the adjacency relationship between hypernodes and hyperedges in an end-to-end manner and thus the task-aware hypergraph is learned.
  arXiv  Detail & Related papers  (2021-06-12T02:07:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.