Understanding Survey Paper Taxonomy about Large Language Models via
Graph Representation Learning
- URL: http://arxiv.org/abs/2402.10409v1
- Date: Fri, 16 Feb 2024 02:21:59 GMT
- Title: Understanding Survey Paper Taxonomy about Large Language Models via
Graph Representation Learning
- Authors: Jun Zhuang, Casey Kennington
- Abstract summary: We develop a method to automatically assign survey papers to a taxonomy.
Our work indicates that leveraging graph structure information on co-category graphs can significantly outperform the language models.
- Score: 2.88268082568407
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: As new research on Large Language Models (LLMs) continues, it is difficult to
keep up with new research and models. To help researchers synthesize the new
research many have written survey papers, but even those have become numerous.
In this paper, we develop a method to automatically assign survey papers to a
taxonomy. We collect the metadata of 144 LLM survey papers and explore three
paradigms to classify papers within the taxonomy. Our work indicates that
leveraging graph structure information on co-category graphs can significantly
outperform the language models in two paradigms; pre-trained language models'
fine-tuning and zero-shot/few-shot classifications using LLMs. We find that our
model surpasses an average human recognition level and that fine-tuning LLMs
using weak labels generated by a smaller model, such as the GCN in this study,
can be more effective than using ground-truth labels, revealing the potential
of weak-to-strong generalization in the taxonomy classification task.
Related papers
- Why do you cite? An investigation on citation intents and decision-making classification processes [1.7812428873698407]
This study emphasizes the importance of trustfully classifying citation intents.
We present a study utilizing advanced Ensemble Strategies for Citation Intent Classification (CIC)
One of our models sets as a new state-of-the-art (SOTA) with an 89.46% Macro-F1 score on the SciCite benchmark.
arXiv Detail & Related papers (2024-07-18T09:29:33Z) - Small Language Models are Good Too: An Empirical Study of Zero-Shot Classification [4.4467858321751015]
We benchmark language models from 77M to 40B parameters using different architectures and scoring functions.
Our findings reveal that small models can effectively classify texts, getting on par with or surpassing their larger counterparts.
This research underscores the notion that bigger isn't always better, suggesting that resource-efficient small models may offer viable solutions for specific data classification challenges.
arXiv Detail & Related papers (2024-04-17T07:10:28Z) - Exploring the Potential of Large Language Models in Graph Generation [51.046188600990014]
Graph generation requires large language models (LLMs) to generate graphs with given properties.
This paper explores the abilities of LLMs for graph generation with systematical task designs and experiments.
Our evaluations demonstrate that LLMs, particularly GPT-4, exhibit preliminary abilities in graph generation tasks.
arXiv Detail & Related papers (2024-03-21T12:37:54Z) - Distilling Large Language Models for Text-Attributed Graph Learning [17.64577949081361]
Text-Attributed Graphs (TAGs) are graphs of connected textual documents.
Graph models can efficiently learn TAGs, but their training heavily relies on human-annotated labels.
Large language models (LLMs) have recently demonstrated remarkable capabilities in few-shot and zero-shot TAG learning.
arXiv Detail & Related papers (2024-02-19T10:31:53Z) - A Survey of Graph Meets Large Language Model: Progress and Future Directions [38.63080573825683]
Large Language Models (LLMs) have achieved tremendous success in various domains.
LLMs have been leveraged in graph-related tasks to surpass traditional Graph Neural Networks (GNNs) based methods.
arXiv Detail & Related papers (2023-11-21T07:22:48Z) - Disentangled Representation Learning with Large Language Models for
Text-Attributed Graphs [57.052160123387104]
We present the Disentangled Graph-Text Learner (DGTL) model, which is able to enhance the reasoning and predicting capabilities of LLMs for TAGs.
Our proposed DGTL model incorporates graph structure information through tailored disentangled graph neural network (GNN) layers.
Experimental evaluations demonstrate the effectiveness of the proposed DGTL model on achieving superior or comparable performance over state-of-the-art baselines.
arXiv Detail & Related papers (2023-10-27T14:00:04Z) - Language models are weak learners [71.33837923104808]
We show that prompt-based large language models can operate effectively as weak learners.
We incorporate these models into a boosting approach, which can leverage the knowledge within the model to outperform traditional tree-based boosting.
Results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning pipelines.
arXiv Detail & Related papers (2023-06-25T02:39:19Z) - Attention is Not Always What You Need: Towards Efficient Classification
of Domain-Specific Text [1.1508304497344637]
For large-scale IT corpora with hundreds of classes organized in a hierarchy, the task of accurate classification of classes at the higher level in the hierarchies is crucial.
In the business world, an efficient and explainable ML model is preferred over an expensive black-box model, especially if the performance increase is marginal.
Despite the widespread use of PLMs, there is a lack of a clear and well-justified need to as why these models are being employed for domain-specific text classification.
arXiv Detail & Related papers (2023-03-31T03:17:23Z) - Large Language Models Are Latent Variable Models: Explaining and Finding
Good Demonstrations for In-Context Learning [104.58874584354787]
In recent years, pre-trained large language models (LLMs) have demonstrated remarkable efficiency in achieving an inference-time few-shot learning capability known as in-context learning.
This study aims to examine the in-context learning phenomenon through a Bayesian lens, viewing real-world LLMs as latent variable models.
arXiv Detail & Related papers (2023-01-27T18:59:01Z) - A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic,
and Multimodal [57.8455911689554]
Knowledge graph reasoning (KGR) aims to deduce new facts from existing facts based on mined logic rules underlying knowledge graphs (KGs)
It has been proven to significantly benefit the usage of KGs in many AI applications, such as question answering, recommendation systems, and etc.
arXiv Detail & Related papers (2022-12-12T08:40:04Z) - Revisiting Self-Training for Few-Shot Learning of Language Model [61.173976954360334]
Unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model.
In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM.
arXiv Detail & Related papers (2021-10-04T08:51:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.