CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations

• URL: http://arxiv.org/abs/2204.07142v1
• Date: Thu, 14 Apr 2022 17:54:46 GMT
• Title: CLUES: A Benchmark for Learning Classifiers using Natural Language Explanations
• Authors: Rakesh R Menon, Sayan Ghosh, Shashank Srivastava
• Abstract summary: Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task. In contrast, humans have ability to learn new concepts from language. We introduce CLUES, benchmark for learning using natural language ExplanationS. CLUES consists of 36 real-world and 144 synthetic classification tasks.
• Score: 12.278877764015725
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Supervised learning has traditionally focused on inductive learning by observing labeled examples of a task. In contrast, humans have the ability to learn new concepts from language. Here, we explore training zero-shot classifiers for structured data purely from language. For this, we introduce CLUES, a benchmark for Classifier Learning Using natural language ExplanationS, consisting of a range of classification tasks over structured data along with natural language supervision in the form of explanations. CLUES consists of 36 real-world and 144 synthetic classification tasks. It contains crowdsourced explanations describing real-world tasks from multiple teachers and programmatically generated explanations for the synthetic tasks. To model the influence of explanations in classifying an example, we develop ExEnt, an entailment-based model that learns classifiers using explanations. ExEnt generalizes up to 18% better (relative) on novel tasks than a baseline that does not use explanations. We delineate key challenges for automated learning from explanations, addressing which can lead to progress on CLUES in the future. Code and datasets are available at: https://clues-benchmark.github.io.

Related papers

• Leveraging Multiple Teachers for Test-Time Adaptation of Language-Guided Classifiers [21.55294900536358]
  We introduce TALC, a framework that adapts a language-guided classifier for a new task during inference when provided with explanations from multiple teachers and unlabeled test examples. Our results show that TALC consistently outperforms a competitive baseline from prior work by an impressive 9.3% (relative improvement)
  arXiv  Detail & Related papers  (2023-11-13T18:28:25Z)
• Language models are weak learners [71.33837923104808]
  We show that prompt-based large language models can operate effectively as weak learners. We incorporate these models into a boosting approach, which can leverage the knowledge within the model to outperform traditional tree-based boosting. Results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning pipelines.
  arXiv  Detail & Related papers  (2023-06-25T02:39:19Z)
• MaNtLE: Model-agnostic Natural Language Explainer [9.43206883360088]
  We introduce MaNtLE, a model-agnostic natural language explainer that analyzes multiple classifier predictions. MaNtLE uses multi-task training on thousands of synthetic classification tasks to generate faithful explanations. Simulated user studies indicate that, on average, MaNtLE-generated explanations are at least 11% more faithful compared to LIME and Anchors explanations.
  arXiv  Detail & Related papers  (2023-05-22T12:58:06Z)
• CodeKGC: Code Language Model for Generative Knowledge Graph Construction [46.220237225553234]
  Large generative language model trained on structured data such as code has demonstrated impressive capability in understanding natural language for structural prediction and reasoning tasks. We develop schema-aware prompts that effectively utilize the semantic structure within the knowledge graph. Experimental results indicate that the proposed approach can obtain better performance on benchmark datasets compared with baselines.
  arXiv  Detail & Related papers  (2023-04-18T15:12:34Z)
• Python Code Generation by Asking Clarification Questions [57.63906360576212]
  In this work, we introduce a novel and more realistic setup for this task. We hypothesize that the under-specification of a natural language description can be resolved by asking clarification questions. We collect and introduce a new dataset named CodeClarQA containing pairs of natural language descriptions and code with created synthetic clarification questions and answers.
  arXiv  Detail & Related papers  (2022-12-19T22:08:36Z)
• Zero-Shot Classification by Logical Reasoning on Natural Language Explanations [56.42922904777717]
  We propose the framework CLORE (Classification by LOgical Reasoning on Explanations) CLORE parses explanations into logical structures and then explicitly reasons along thess structures on the input to produce a classification score. We also demonstrate that our framework can be extended to zero-shot classification on visual modality.
  arXiv  Detail & Related papers  (2022-11-07T01:05:11Z)
• Explaining Patterns in Data with Language Models via Interpretable Autoprompting [143.4162028260874]
  We introduce interpretable autoprompting (iPrompt), an algorithm that generates a natural-language string explaining the data. iPrompt can yield meaningful insights by accurately finding groundtruth dataset descriptions. Experiments with an fMRI dataset show the potential for iPrompt to aid in scientific discovery.
  arXiv  Detail & Related papers  (2022-10-04T18:32:14Z)
• Joint Language Semantic and Structure Embedding for Knowledge Graph Completion [66.15933600765835]
  We propose to jointly embed the semantics in the natural language description of the knowledge triplets with their structure information. Our method embeds knowledge graphs for the completion task via fine-tuning pre-trained language models. Our experiments on a variety of knowledge graph benchmarks have demonstrated the state-of-the-art performance of our method.
  arXiv  Detail & Related papers  (2022-09-19T02:41:02Z)
• ALICE: Active Learning with Contrastive Natural Language Explanations [69.03658685761538]
  We propose Active Learning with Contrastive Explanations (ALICE) to improve data efficiency in learning. ALICE learns to first use active learning to select the most informative pairs of label classes to elicit contrastive natural language explanations. It extracts knowledge from these explanations using a semantically extracted knowledge.
  arXiv  Detail & Related papers  (2020-09-22T01:02:07Z)
• Teaching Machine Comprehension with Compositional Explanations [32.82449839424392]
  We focus on "teaching" machines reading comprehension, using a small number of semi-structured explanations. We use learnable neural modules and soft logic to handle linguistic variation and overcome sparse coverage. On the SQuAD dataset, our proposed method achieves 70.14% F1 score with supervision from 26 explanations, comparable to plain supervised learning.
  arXiv  Detail & Related papers  (2020-05-02T11:54:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.