PEACH: Pretrained-embedding Explanation Across Contextual and Hierarchical Structure

• URL: http://arxiv.org/abs/2404.13645v1
• Date: Sun, 21 Apr 2024 12:41:02 GMT
• Title: PEACH: Pretrained-embedding Explanation Across Contextual and Hierarchical Structure
• Authors: Feiqi Cao, Caren Han, Hyunsuk Chung,
• Abstract summary: PEACH can explain how text-based documents are classified by using any pretrained contextual embeddings in a tree-based human-interpretable manner. We perform a comprehensive analysis of several contextual embeddings on nine different NLP text classification benchmarks. We show the utility of explanations by visualising the feature selection and important trend of text classification via human-interpretable word-cloud-based trees.
• Score: 3.9677082086241433
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we propose a novel tree-based explanation technique, PEACH (Pretrained-embedding Explanation Across Contextual and Hierarchical Structure), that can explain how text-based documents are classified by using any pretrained contextual embeddings in a tree-based human-interpretable manner. Note that PEACH can adopt any contextual embeddings of the PLMs as a training input for the decision tree. Using the proposed PEACH, we perform a comprehensive analysis of several contextual embeddings on nine different NLP text classification benchmarks. This analysis demonstrates the flexibility of the model by applying several PLM contextual embeddings, its attribute selections, scaling, and clustering methods. Furthermore, we show the utility of explanations by visualising the feature selection and important trend of text classification via human-interpretable word-cloud-based trees, which clearly identify model mistakes and assist in dataset debugging. Besides interpretability, PEACH outperforms or is similar to those from pretrained models.

Related papers

• Pointer-Guided Pre-Training: Infusing Large Language Models with Paragraph-Level Contextual Awareness [3.2925222641796554]
  "pointer-guided segment ordering" (SO) is a novel pre-training technique aimed at enhancing the contextual understanding of paragraph-level text representations. Our experiments show that pointer-guided pre-training significantly enhances the model's ability to understand complex document structures.
  arXiv  Detail & Related papers  (2024-06-06T15:17:51Z)
• Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs [19.073560504913356]
  The line between human-crafted and machine-generated texts has become increasingly blurred. This paper delves into the inquiry of identifying discernible and unique linguistic properties in texts that were written by humans.
  arXiv  Detail & Related papers  (2024-02-16T11:20:30Z)
• Answer is All You Need: Instruction-following Text Embedding via Answering the Question [41.727700155498546]
  This paper offers a new viewpoint, which treats the instruction as a question about the input text and encodes the expected answers to obtain the representation accordingly. Specifically, we propose InBedder that instantiates this embed-via-answering idea by only fine-tuning language models on abstractive question answering tasks.
  arXiv  Detail & Related papers  (2024-02-15T01:02:41Z)
• Scalable Learning of Latent Language Structure With Logical Offline Cycle Consistency [71.42261918225773]
  Conceptually, LOCCO can be viewed as a form of self-learning where the semantic being trained is used to generate annotations for unlabeled text. As an added bonus, the annotations produced by LOCCO can be trivially repurposed to train a neural text generation model.
  arXiv  Detail & Related papers  (2023-05-31T16:47:20Z)
• CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model [55.321010757641524]
  We introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP. We scale CLIP4STR in terms of the model size, pre-training data, and training data, achieving state-of-the-art performance on 11 STR benchmarks.
  arXiv  Detail & Related papers  (2023-05-23T12:51:20Z)
• A Multi-Grained Self-Interpretable Symbolic-Neural Model For Single/Multi-Labeled Text Classification [29.075766631810595]
  We propose a Symbolic-Neural model that can learn to explicitly predict class labels of text spans from a constituency tree. As the structured language model learns to predict constituency trees in a self-supervised manner, only raw texts and sentence-level labels are required as training data. Our experiments demonstrate that our approach could achieve good prediction accuracy in downstream tasks.
  arXiv  Detail & Related papers  (2023-03-06T03:25:43Z)
• Autoregressive Structured Prediction with Language Models [73.11519625765301]
  We describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at.
  arXiv  Detail & Related papers  (2022-10-26T13:27:26Z)
• Textual Entailment Recognition with Semantic Features from Empirical Text Representation [60.31047947815282]
  A text entails a hypothesis if and only if the true value of the hypothesis follows the text. In this paper, we propose a novel approach to identifying the textual entailment relationship between text and hypothesis. We employ an element-wise Manhattan distance vector-based feature that can identify the semantic entailment relationship between the text-hypothesis pair.
  arXiv  Detail & Related papers  (2022-10-18T10:03:51Z)
• A Unified Understanding of Deep NLP Models for Text Classification [88.35418976241057]
  We have developed a visual analysis tool, DeepNLPVis, to enable a unified understanding of NLP models for text classification. The key idea is a mutual information-based measure, which provides quantitative explanations on how each layer of a model maintains the information of input words in a sample. A multi-level visualization, which consists of a corpus-level, a sample-level, and a word-level visualization, supports the analysis from the overall training set to individual samples.
  arXiv  Detail & Related papers  (2022-06-19T08:55:07Z)
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing [78.8500633981247]
  This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning" Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly.
  arXiv  Detail & Related papers  (2021-07-28T18:09:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.