Contextualizing the Limits of Model & Evaluation Dataset Curation on Semantic Similarity Classification Tasks

• URL: http://arxiv.org/abs/2311.04927v1
• Date: Fri, 3 Nov 2023 17:12:07 GMT
• Title: Contextualizing the Limits of Model & Evaluation Dataset Curation on Semantic Similarity Classification Tasks
• Authors: Daniel Theron
• Abstract summary: This paper demonstrates how the limitations of pre-trained models and open evaluation datasets factor into assessing the performance of binary semantic similarity classification tasks. As (1) end-user-facing documentation around the curation of these datasets and pre-trained model training regimes is often not easily accessible and (2) given the lower friction and higher demand to quickly deploy such systems in real-world contexts, our study reinforces prior work showing performance disparities across datasets, embedding techniques and distance metrics.
• Score: 1.8130068086063336
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper demonstrates how the limitations of pre-trained models and open evaluation datasets factor into assessing the performance of binary semantic similarity classification tasks. As (1) end-user-facing documentation around the curation of these datasets and pre-trained model training regimes is often not easily accessible and (2) given the lower friction and higher demand to quickly deploy such systems in real-world contexts, our study reinforces prior work showing performance disparities across datasets, embedding techniques and distance metrics, while highlighting the importance of understanding how data is collected, curated and analyzed in semantic similarity classification.

Related papers

• Contextuality Helps Representation Learning for Generalized Category Discovery [5.885208652383516]
  This paper introduces a novel approach to Generalized Category Discovery (GCD) by leveraging the concept of contextuality. Our model integrates two levels of contextuality: instance-level, where nearest-neighbor contexts are utilized for contrastive learning, and cluster-level, employing contrastive learning. The integration of the contextual information effectively improves the feature learning and thereby the classification accuracy of all categories.
  arXiv  Detail & Related papers  (2024-07-29T07:30:41Z)
• Detecting Statements in Text: A Domain-Agnostic Few-Shot Solution [1.3654846342364308]
  State-of-the-art approaches usually involve fine-tuning models on large annotated datasets, which are costly to produce. We propose and release a qualitative and versatile few-shot learning methodology as a common paradigm for any claim-based textual classification task. We illustrate this methodology in the context of three tasks: climate change contrarianism detection, topic/stance classification and depression-relates symptoms detection.
  arXiv  Detail & Related papers  (2024-05-09T12:03:38Z)
• Distilled Datamodel with Reverse Gradient Matching [74.75248610868685]
  We introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages. Our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.
  arXiv  Detail & Related papers  (2024-04-22T09:16:14Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Estimating class separability of text embeddings with persistent homology [1.9956517534421363]
  This paper introduces an unsupervised method to estimate the class separability of text datasets from a topological point of view. We show how this technique can be applied to detect when the training process stops improving the separability of the embeddings. Our results, validated across binary and multi-class text classification tasks, show that the proposed method's estimates of class separability align with those obtained from supervised methods.
  arXiv  Detail & Related papers  (2023-05-24T10:58:09Z)
• A classification performance evaluation measure considering data separability [6.751026374812737]
  We propose a new separability measure--the rate of separability (RS)--based on the data coding rate. We demonstrate the positive correlation between the proposed measure and recognition accuracy in a multi-task scenario constructed from a real dataset.
  arXiv  Detail & Related papers  (2022-11-10T09:18:26Z)
• Systematic Evaluation of Predictive Fairness [60.0947291284978]
  Mitigating bias in training on biased datasets is an important open problem. We examine the performance of various debiasing methods across multiple tasks. We find that data conditions have a strong influence on relative model performance.
  arXiv  Detail & Related papers  (2022-10-17T05:40:13Z)
• Learning from Temporal Spatial Cubism for Cross-Dataset Skeleton-based Action Recognition [88.34182299496074]
  Action labels are only available on a source dataset, but unavailable on a target dataset in the training stage. We utilize a self-supervision scheme to reduce the domain shift between two skeleton-based action datasets. By segmenting and permuting temporal segments or human body parts, we design two self-supervised learning classification tasks.
  arXiv  Detail & Related papers  (2022-07-17T07:05:39Z)
• Conditional Supervised Contrastive Learning for Fair Text Classification [59.813422435604025]
  We study learning fair representations that satisfy a notion of fairness known as equalized odds for text classification via contrastive learning. Specifically, we first theoretically analyze the connections between learning representations with a fairness constraint and conditional supervised contrastive objectives.
  arXiv  Detail & Related papers  (2022-05-23T17:38:30Z)
• CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems [121.78477833009671]
  We investigate the performance of different summarization models under a cross-dataset setting. A comprehensive study of 11 representative summarization systems on 5 datasets from different domains reveals the effect of model architectures and generation ways.
  arXiv  Detail & Related papers  (2020-10-11T02:19:15Z)
• Contrastive estimation reveals topic posterior information to linear models [38.80336134485453]
  Contrastive learning is an approach to representation learning that utilizes naturally occurring similar and dissimilar pairs of data points to find useful embeddings of data. We prove that contrastive learning is capable of recovering a representation of documents that reveals their underlying topic posterior information to linear models.
  arXiv  Detail & Related papers  (2020-03-04T18:20:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.