Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes

• URL: http://arxiv.org/abs/2104.08481v1
• Date: Sat, 17 Apr 2021 08:16:49 GMT
• Title: Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes
• Authors: Ofer Sabo, Yanai Elazar, Yoav Goldberg, Ido Dagan
• Abstract summary: We propose a novel methodology for deriving more realistic few-shot test data from available datasets for supervised RC. This yields a new challenging benchmark for FSL RC, on which state of the art models show poor performance. We propose a novel classification scheme, in which the NOTA category is represented as learned vectors.
• Score: 57.34346419239118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We explore Few-Shot Learning (FSL) for Relation Classification (RC). Focusing on the realistic scenario of FSL, in which a test instance might not belong to any of the target categories (none-of-the-above, aka NOTA), we first revisit the recent popular dataset structure for FSL, pointing out its unrealistic data distribution. To remedy this, we propose a novel methodology for deriving more realistic few-shot test data from available datasets for supervised RC, and apply it to the TACRED dataset. This yields a new challenging benchmark for FSL RC, on which state of the art models show poor performance. Next, we analyze classification schemes within the popular embedding-based nearest-neighbor approach for FSL, with respect to constraints they impose on the embedding space. Triggered by this analysis we propose a novel classification scheme, in which the NOTA category is represented as learned vectors, shown empirically to be an appealing option for FSL.

Related papers

• High-Performance Few-Shot Segmentation with Foundation Models: An Empirical Study [64.06777376676513]
  We develop a few-shot segmentation (FSS) framework based on foundation models. To be specific, we propose a simple approach to extract implicit knowledge from foundation models to construct coarse correspondence. Experiments on two widely used datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-09-10T08:04:11Z)
• EOL: Transductive Few-Shot Open-Set Recognition by Enhancing Outlier Logits [16.081748213657825]
  In Few-Shot Learning, models are trained to recognise unseen objects from a query set, given a few labelled examples from a support set. In this work, we explore the more nuanced and practical challenge of Open-Set Few-Shot Recognition.
  arXiv  Detail & Related papers  (2024-08-04T15:00:22Z)
• A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification [51.35500308126506]
  Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels. We study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types.
  arXiv  Detail & Related papers  (2024-07-16T23:17:36Z)
• Co-training for Low Resource Scientific Natural Language Inference [65.37685198688538]
  We propose a novel co-training method that assigns weights based on the training dynamics of the classifiers to the distantly supervised labels. By assigning importance weights instead of filtering out examples based on an arbitrary threshold on the predicted confidence, we maximize the usage of automatically labeled data. The proposed method obtains an improvement of 1.5% in Macro F1 over the distant supervision baseline, and substantial improvements over several other strong SSL baselines.
  arXiv  Detail & Related papers  (2024-06-20T18:35:47Z)
• PatchMix Augmentation to Identify Causal Features in Few-shot Learning [55.64873998196191]
  Few-shot learning aims to transfer knowledge learned from base with sufficient categories labelled data to novel categories with scarce known information. We propose a novel data augmentation strategy dubbed as PatchMix that can break this spurious dependency. We show that such an augmentation mechanism, different from existing ones, is able to identify the causal features.
  arXiv  Detail & Related papers  (2022-11-29T08:41:29Z)
• GCT: Graph Co-Training for Semi-Supervised Few-Shot Learning [46.07923908746946]
  We propose a novel label prediction method, Isolated Graph Learning (IGL) IGL introduces the Laplacian operator to encode the raw data to graph space, which helps reduce the dependence on features when classifying. Second, we propose Graph Co-Training (GCT) to tackle this challenge from a multi-modal fusion perspective.
  arXiv  Detail & Related papers  (2022-03-15T09:13:35Z)
• A Strong Baseline for Semi-Supervised Incremental Few-Shot Learning [54.617688468341704]
  Few-shot learning aims to learn models that generalize to novel classes with limited training samples. We propose a novel paradigm containing two parts: (1) a well-designed meta-training algorithm for mitigating ambiguity between base and novel classes caused by unreliable pseudo labels and (2) a model adaptation mechanism to learn discriminative features for novel classes while preserving base knowledge using few labeled and all the unlabeled data.
  arXiv  Detail & Related papers  (2021-10-21T13:25:52Z)
• A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification [38.68079253627819]
  Our benchmark consists of two fine-grained classification datasets obtained by sampling classes from the Aves and Fungi taxonomy. We find that recently proposed SSL methods provide significant benefits, and can effectively use out-of-class data to improve performance when deep networks are trained from scratch. Our work suggests that semi-supervised learning with experts on realistic datasets may require different strategies than those currently prevalent in the literature.
  arXiv  Detail & Related papers  (2021-04-01T17:59:41Z)
• Match Them Up: Visually Explainable Few-shot Image Classification [27.867833878756553]
  Few-shot learning is usually based on an assumption that the pre-trained knowledge can be obtained from base (seen) categories and can be well transferred to novel (unseen) categories. In this paper, we reveal a new way to perform FSL for image classification, using visual representations from the backbone model and weights generated by a newly-emerged explainable classifier. Experimental results prove that the proposed method can achieve both good accuracy and satisfactory explainability on three mainstream datasets.
  arXiv  Detail & Related papers  (2020-11-25T05:47:35Z)
• Few-shot Image Recognition with Manifolds [10.312968200748116]
  We extend the traditional few-shot learning problem to the situation when the source-domain data is not accessible. Because of limited training data, we propose a non-parametric approach by assuming that all the class prototypes are structurally arranged on a manifold. During classification, we again exploit the structural arrangement of the categories by inducing a Markov chain on the graph constructed with the class prototypes.
  arXiv  Detail & Related papers  (2020-10-22T21:57:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.