RaTE: a Reproducible automatic Taxonomy Evaluation by Filling the Gap

• URL: http://arxiv.org/abs/2307.09706v1
• Date: Wed, 19 Jul 2023 01:37:31 GMT
• Title: RaTE: a Reproducible automatic Taxonomy Evaluation by Filling the Gap
• Authors: Tianjian Gao and Phillipe Langlais
• Abstract summary: We argue that automatic taxonomy evaluation (ATE) is just as important as taxonomy construction. We propose RaTE, an automatic label-free taxonomy scoring procedure, which relies on a large pre-trained language model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Taxonomies are an essential knowledge representation, yet most studies on automatic taxonomy construction (ATC) resort to manual evaluation to score proposed algorithms. We argue that automatic taxonomy evaluation (ATE) is just as important as taxonomy construction. We propose RaTE, an automatic label-free taxonomy scoring procedure, which relies on a large pre-trained language model. We apply our evaluation procedure to three state-of-the-art ATC algorithms with which we built seven taxonomies from the Yelp domain, and show that 1) RaTE correlates well with human judgments and 2) artificially degrading a taxonomy leads to decreasing RaTE score.

Related papers

• ASTAGEN: Empirical Evaluation of Automated SATD Taxonomy Generation with LLMs [2.287480001913659]
  Self-admitted technical debt (SATD) refers to suboptimal code that degrades software quality.<n>This study presents ASTAGEN, an initial step toward automating SATD taxonomy generation using large language models (LLMs)<n>We evaluate ASTAGEN on SATD datasets from three domains: quantum software, smart contracts, and machine learning.
  arXiv  Detail & Related papers  (2025-06-11T10:59:57Z)
• Taxonomy-Aware Evaluation of Vision-Language Models [48.285819827561625]
  We propose a framework for evaluating unconstrained text predictions, such as those generated from a vision-language model, against a taxonomy. Specifically, we propose the use of hierarchical precision and recall measures to assess the level of correctness and specificity of predictions with regard to a taxonomy.
  arXiv  Detail & Related papers  (2025-04-07T19:46:59Z)
• Do I look like a `cat.n.01` to you? A Taxonomy Image Generation Benchmark [63.97125827026949]
  This paper explores the feasibility of using text-to-image models in a zero-shot setup to generate images for taxonomy concepts. A benchmark is proposed that assesses models' abilities to understand taxonomy concepts and generate relevant, high-quality images. The 12 models are evaluated using 9 novel taxonomy-related text-to-image metrics and human feedback.
  arXiv  Detail & Related papers  (2025-03-13T13:37:54Z)
• Creating a Fine Grained Entity Type Taxonomy Using LLMs [0.0]
  This study investigates the potential of GPT-4 and its advanced iteration, GPT-4 Turbo, in autonomously developing a detailed entity type taxonomy. Our objective is to construct a comprehensive taxonomy, starting from a broad classification of entity types. This classification is then progressively refined through iterative prompting techniques, leveraging GPT-4's internal knowledge base.
  arXiv  Detail & Related papers  (2024-02-19T21:32:19Z)
• Chain-of-Layer: Iteratively Prompting Large Language Models for Taxonomy Induction from Limited Examples [34.88498567698853]
  Chain-of-Layer is an incontext learning framework designed to induct from a given set of entities. We show that Chain-of-Layer achieves state-of-the-art performance on four real-world benchmarks.
  arXiv  Detail & Related papers  (2024-02-12T03:05:54Z)
• Using Zero-shot Prompting in the Automatic Creation and Expansion of Topic Taxonomies for Tagging Retail Banking Transactions [0.0]
  This work presents an unsupervised method for constructing and expanding topic using instruction-based fine-tuned LLMs (Large Language Models) To expand an existing taxonomy with new terms, we use zero-shot prompting to find out where to add new nodes. We use the resulting tags to assign tags that characterize merchants from a retail bank dataset.
  arXiv  Detail & Related papers  (2024-01-08T00:27:16Z)
• ProTeCt: Prompt Tuning for Taxonomic Open Set Classification [59.59442518849203]
  Few-shot adaptation methods do not fare well in the taxonomic open set (TOS) setting. We propose a prompt tuning technique that calibrates the hierarchical consistency of model predictions. A new Prompt Tuning for Hierarchical Consistency (ProTeCt) technique is then proposed to calibrate classification across label set granularities.
  arXiv  Detail & Related papers  (2023-06-04T02:55:25Z)
• Insert or Attach: Taxonomy Completion via Box Embedding [75.69894194912595]
  Previous approaches embed concepts as vectors in Euclidean space, which makes it difficult to model asymmetric relations in taxonomy. We develop a framework, TaxBox, that leverages box containment and center closeness to design two specialized geometric scorers within the box embedding space. These scorers are tailored for insertion and attachment operations and can effectively capture intrinsic relationships between concepts.
  arXiv  Detail & Related papers  (2023-05-18T14:34:58Z)
• CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification [70.554573538777]
  We argue for hierarchical evaluation of the predictions of neural LMTC models. We describe a structural issue in the representation of the structured label space in prior art. We propose a set of metrics for hierarchical evaluation using the depth-based representation.
  arXiv  Detail & Related papers  (2021-09-10T13:09:12Z)
• Large-scale Taxonomy Induction Using Entity and Word Embeddings [13.30719395448771]
  We propose TIEmb, an approach for automatic subsumption extraction from knowledge using entity and text embeddings. We apply the approach on the WebIsA database, a database of classes subsumption relations extracted from the large portion of Wide Web, to extract hierarchies in the Person and Place domain.
  arXiv  Detail & Related papers  (2021-05-04T05:53:12Z)
• Can Taxonomy Help? Improving Semantic Question Matching using Question Taxonomy [37.57300969050908]
  We propose a hybrid technique for semantic question matching. It uses our proposed two-layered taxonomy for English questions by augmenting state-of-the-art deep learning models with question classes obtained from a deep learning based question.
  arXiv  Detail & Related papers  (2021-01-20T16:23:04Z)
• Octet: Online Catalog Taxonomy Enrichment with Self-Supervision [67.26804972901952]
  We present a self-supervised end-to-end framework, Octet for Online Catalog EnrichmenT. We propose to train a sequence labeling model for term extraction and employ graph neural networks (GNNs) to capture the taxonomy structure. Octet enriches an online catalog in production to 2 times larger in the open-world evaluation.
  arXiv  Detail & Related papers  (2020-06-18T04:53:07Z)
• STEAM: Self-Supervised Taxonomy Expansion with Mini-Paths [53.45704816829921]
  We propose a self-supervised taxonomy expansion model named STEAM. STEAM generates natural self-supervision signals, and formulates a node attachment prediction task. Experiments show STEAM outperforms state-of-the-art methods for taxonomy expansion by 11.6% in accuracy and 7.0% in mean reciprocal rank.
  arXiv  Detail & Related papers  (2020-06-18T00:32:53Z)
• TaxoExpan: Self-supervised Taxonomy Expansion with Position-Enhanced Graph Neural Network [62.12557274257303]
  Taxonomies consist of machine-interpretable semantics and provide valuable knowledge for many web applications. We propose a novel self-supervised framework, named TaxoExpan, which automatically generates a set of query concept, anchor concept> pairs from the existing taxonomy as training data. We develop two innovative techniques in TaxoExpan: (1) a position-enhanced graph neural network that encodes the local structure of an anchor concept in the existing taxonomy, and (2) a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data.
  arXiv  Detail & Related papers  (2020-01-26T21:30:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.