Analyzing Text Representations by Measuring Task Alignment

• URL: http://arxiv.org/abs/2305.19747v1
• Date: Wed, 31 May 2023 11:20:48 GMT
• Title: Analyzing Text Representations by Measuring Task Alignment
• Authors: Cesar Gonzalez-Gutierrez, Audi Primadhanty, Francesco Cazzaro, Ariadna Quattoni
• Abstract summary: We develop a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity. Our experiments on text classification validate our hypothesis by showing that task alignment can explain the classification performance of a given representation.
• Score: 2.198430261120653
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Textual representations based on pre-trained language models are key, especially in few-shot learning scenarios. What makes a representation good for text classification? Is it due to the geometric properties of the space or because it is well aligned with the task? We hypothesize the second claim. To test it, we develop a task alignment score based on hierarchical clustering that measures alignment at different levels of granularity. Our experiments on text classification validate our hypothesis by showing that task alignment can explain the classification performance of a given representation.

Related papers

• Probing Context Localization of Polysemous Words in Pre-trained Language Model Sub-Layers [12.610445666406898]
  We investigate the degree of contextualization encoded in the fine-grained sub-layer representations of a Pre-trained Language Model (PLM) To identify the main contributions of sub-layers to contextualisation, we first extract the sub-layer representations of polysemous words in minimally different sentence pairs. We also try to empirically localize the strength of contextualization information encoded in these sub-layer representations.
  arXiv  Detail & Related papers  (2024-09-21T10:42:07Z)
• Are we describing the same sound? An analysis of word embedding spaces of expressive piano performance [4.867952721052875]
  We investigate the uncertainty for the domain of characterizations of expressive piano performance. We test five embedding models and their similarity structure for correspondence with the ground truth. The quality of embedding models shows great variability with respect to this task.
  arXiv  Detail & Related papers  (2023-12-31T12:20:03Z)
• Textual Entailment Recognition with Semantic Features from Empirical Text Representation [60.31047947815282]
  A text entails a hypothesis if and only if the true value of the hypothesis follows the text. In this paper, we propose a novel approach to identifying the textual entailment relationship between text and hypothesis. We employ an element-wise Manhattan distance vector-based feature that can identify the semantic entailment relationship between the text-hypothesis pair.
  arXiv  Detail & Related papers  (2022-10-18T10:03:51Z)
• Analyzing Text Representations under Tight Annotation Budgets: Measuring Structural Alignment [2.198430261120653]
  Under tight annotation budgets the choice of data representation is key. We propose a metric that measures the extent to which a given representation is structurally aligned with a task.
  arXiv  Detail & Related papers  (2022-10-11T18:28:19Z)
• Text Revision by On-the-Fly Representation Optimization [76.11035270753757]
  Current state-of-the-art methods formulate these tasks as sequence-to-sequence learning problems. We present an iterative in-place editing approach for text revision, which requires no parallel data. It achieves competitive and even better performance than state-of-the-art supervised methods on text simplification.
  arXiv  Detail & Related papers  (2022-04-15T07:38:08Z)
• Conditional probing: measuring usable information beyond a baseline [103.93673427217527]
  One suggests that a representation encodes a property if probing that representation produces higher accuracy than probing a baseline representation. We propose conditional probing, which explicitly conditions on the information in the baseline. In a case study, we find that after conditioning on non-contextual word embeddings, properties like part-of-speech are accessible at deeper layers of a network.
  arXiv  Detail & Related papers  (2021-09-19T21:56:58Z)
• Exploring Set Similarity for Dense Self-supervised Representation Learning [96.35286140203407]
  We propose to explore textbfset textbfsimilarity (SetSim) for dense self-supervised representation learning. We generalize pixel-wise similarity learning to set-wise one to improve the robustness because sets contain more semantic and structure information. Specifically, by resorting to attentional features of views, we establish corresponding sets, thus filtering out noisy backgrounds that may cause incorrect correspondences.
  arXiv  Detail & Related papers  (2021-07-19T09:38:27Z)
• DirectProbe: Studying Representations without Classifiers [21.23284793831221]
  DirectProbe studies the geometry of a representation by building upon the notion of a version space for a task. Experiments with several linguistic tasks and contextualized embeddings show that, even without training classifiers, DirectProbe can shine light into how an embedding space represents labels.
  arXiv  Detail & Related papers  (2021-04-13T02:40:26Z)
• X-Class: Text Classification with Extremely Weak Supervision [39.25777650619999]
  In this paper, we explore text classification with extremely weak supervision. We propose a novel framework X-Class to realize the adaptive representations. X-Class can rival and even outperform seed-driven weakly supervised methods on 7 benchmark datasets.
  arXiv  Detail & Related papers  (2020-10-24T06:09:51Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)
• Predicting What You Already Know Helps: Provable Self-Supervised Learning [60.27658820909876]
  Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data. We show a mechanism exploiting the statistical connections between certain em reconstruction-based pretext tasks that guarantee to learn a good representation. We prove the linear layer yields small approximation error even for complex ground truth function class.
  arXiv  Detail & Related papers  (2020-08-03T17:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.