Complexity-Guided Curriculum Learning for Text Graphs

• URL: http://arxiv.org/abs/2311.13472v1
• Date: Wed, 22 Nov 2023 15:40:57 GMT
• Title: Complexity-Guided Curriculum Learning for Text Graphs
• Authors: Nidhi Vakil, Hadi Amiri
• Abstract summary: We present a curriculum learning approach that builds on existing knowledge about text and graph complexity formalisms. A novel data scheduler employs "spaced repetition" and complexity formalisms to guide the training process. We demonstrate the effectiveness of the proposed approach on several text graph tasks and graph neural network architectures.
• Score: 18.746490400989487
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Curriculum learning provides a systematic approach to training. It refines training progressively, tailors training to task requirements, and improves generalization through exposure to diverse examples. We present a curriculum learning approach that builds on existing knowledge about text and graph complexity formalisms for training with text graph data. The core part of our approach is a novel data scheduler, which employs "spaced repetition" and complexity formalisms to guide the training process. We demonstrate the effectiveness of the proposed approach on several text graph tasks and graph neural network architectures. The proposed model gains more and uses less data; consistently prefers text over graph complexity indices throughout training, while the best curricula derived from text and graph complexity indices are equally effective; and it learns transferable curricula across GNN models and datasets. In addition, we find that both node-level (local) and graph-level (global) graph complexity indices, as well as shallow and traditional text complexity indices play a crucial role in effective curriculum learning.

Related papers

• Towards Efficient Training of Graph Neural Networks: A Multiscale Approach [20.713913005905297]
  Graph Neural Networks (GNNs) have emerged as a powerful tool for learning and inferring from graph-structured data. We introduce a novel framework for efficient multiscale training of GNNs, designed to integrate information across multiscale representations of a graph.
  arXiv  Detail & Related papers  (2025-03-25T13:52:26Z)
• Graph Learning in the Era of LLMs: A Survey from the Perspective of Data, Models, and Tasks [25.720233631885726]
  integration of Graph Neural Networks (GNNs) and Large Language Models (LLMs) has emerged as a promising technological paradigm. We leverage graph description texts with rich semantic context to fundamentally enhance Data quality. This work serves as a foundational reference for researchers and practitioners looking to advance graph learning methodologies.
  arXiv  Detail & Related papers  (2024-12-17T01:41:17Z)
• SE-GCL: An Event-Based Simple and Effective Graph Contrastive Learning for Text Representation [23.60337935010744]
  We present an event-based, simple, and effective graph contrastive learning (SE-GCL) for text representation. Precisely, we extract event blocks from text and construct internal relation graphs to represent inter-semantic interconnections. In particular, we introduce the concept of an event skeleton for core representation semantics and simplify the typically complex data augmentation techniques.
  arXiv  Detail & Related papers  (2024-12-16T10:53:24Z)
• Node Level Graph Autoencoder: Unified Pretraining for Textual Graph Learning [45.70767623846523]
  We propose a novel unified unsupervised learning autoencoder framework, named Node Level Graph AutoEncoder (NodeGAE) We employ language models as the backbone of the autoencoder, with pretraining on text reconstruction. Our method maintains simplicity in the training process and demonstrates generalizability across diverse textual graphs and downstream tasks.
  arXiv  Detail & Related papers  (2024-08-09T14:57:53Z)
• Hierarchical Compression of Text-Rich Graphs via Large Language Models [63.75293588479027]
  Text-rich graphs are prevalent in data mining contexts like e-commerce and academic graphs. This paper introduces Hierarchical Compression'' (HiCom), a novel method to align the capabilities of LLMs with the structure of text-rich graphs. HiCom can outperform both GNNs and LLM backbones for node classification on e-commerce and citation graphs.
  arXiv  Detail & Related papers  (2024-06-13T07:24:46Z)
• When Graph Data Meets Multimodal: A New Paradigm for Graph Understanding and Reasoning [54.84870836443311]
  The paper presents a new paradigm for understanding and reasoning about graph data by integrating image encoding and multimodal technologies. This approach enables the comprehension of graph data through an instruction-response format, utilizing GPT-4V's advanced capabilities. The study evaluates this paradigm on various graph types, highlighting the model's strengths and weaknesses, particularly in Chinese OCR performance and complex reasoning tasks.
  arXiv  Detail & Related papers  (2023-12-16T08:14:11Z)
• Efficient End-to-end Language Model Fine-tuning on Graphs [21.23522552579571]
  Learning from Text-Attributed Graphs (TAGs) has attracted significant attention due to its wide range of real-world applications. We introduce LEADING, a novel and efficient approach for end-to-end fine-tuning of language models on TAGs. Our proposed approach demonstrates superior performance, achieving state-of-the-art (SOTA) results on the ogbn-arxiv leaderboard.
  arXiv  Detail & Related papers  (2023-12-07T22:35:16Z)
• GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs [22.282756544376493]
  We develop a novel Graph-Centric Language model, GRENADE, to solve the problem of self-supervised representation learning on text-attributed graphs. GRENADE exploits the synergistic effect of both pre-trained language model and graph neural network by optimizing with two specialized self-supervised learning algorithms. The proposed graph-centric self-supervised learning algorithms effectively help GRENADE to capture informative textual semantics as well as structural context information on text-attributed graphs.
  arXiv  Detail & Related papers  (2023-10-23T17:18:35Z)
• GraphGPT: Graph Instruction Tuning for Large Language Models [27.036935149004726]
  Graph Neural Networks (GNNs) have evolved to understand graph structures. To enhance robustness, self-supervised learning (SSL) has become a vital tool for data augmentation. Our research tackles this by advancing graph model generalization in zero-shot learning environments.
  arXiv  Detail & Related papers  (2023-10-19T06:17:46Z)
• GraphGLOW: Universal and Generalizable Structure Learning for Graph Neural Networks [72.01829954658889]
  This paper introduces the mathematical definition of this novel problem setting. We devise a general framework that coordinates a single graph-shared structure learner and multiple graph-specific GNNs. The well-trained structure learner can directly produce adaptive structures for unseen target graphs without any fine-tuning.
  arXiv  Detail & Related papers  (2023-06-20T03:33:22Z)
• PRODIGY: Enabling In-context Learning Over Graphs [112.19056551153454]
  In-context learning is the ability of a pretrained model to adapt to novel and diverse downstream tasks. We develop PRODIGY, the first pretraining framework that enables in-context learning over graphs.
  arXiv  Detail & Related papers  (2023-05-21T23:16:30Z)
• State of the Art and Potentialities of Graph-level Learning [54.68482109186052]
  Graph-level learning has been applied to many tasks including comparison, regression, classification, and more. Traditional approaches to learning a set of graphs rely on hand-crafted features, such as substructures. Deep learning has helped graph-level learning adapt to the growing scale of graphs by extracting features automatically and encoding graphs into low-dimensional representations.
  arXiv  Detail & Related papers  (2023-01-14T09:15:49Z)
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph [53.70520466556453]
  We propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow. In addition, a progressive learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph.
  arXiv  Detail & Related papers  (2021-05-06T12:20:41Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.