L2R2: Leveraging Ranking for Abductive Reasoning
- URL: http://arxiv.org/abs/2005.11223v2
- Date: Tue, 14 Sep 2021 10:59:25 GMT
- Title: L2R2: Leveraging Ranking for Abductive Reasoning
- Authors: Yunchang Zhu, Liang Pang, Yanyan Lan, Xueqi Cheng
- Abstract summary: The abductive natural language inference task ($alpha$NLI) is proposed to evaluate the abductive reasoning ability of a learning system.
A novel $L2R2$ approach is proposed under the learning-to-rank framework.
Experiments on the ART dataset reach the state-of-the-art in the public leaderboard.
- Score: 65.40375542988416
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The abductive natural language inference task ($\alpha$NLI) is proposed to
evaluate the abductive reasoning ability of a learning system. In the
$\alpha$NLI task, two observations are given and the most plausible hypothesis
is asked to pick out from the candidates. Existing methods simply formulate it
as a classification problem, thus a cross-entropy log-loss objective is used
during training. However, discriminating true from false does not measure the
plausibility of a hypothesis, for all the hypotheses have a chance to happen,
only the probabilities are different. To fill this gap, we switch to a ranking
perspective that sorts the hypotheses in order of their plausibilities. With
this new perspective, a novel $L2R^2$ approach is proposed under the
learning-to-rank framework. Firstly, training samples are reorganized into a
ranking form, where two observations and their hypotheses are treated as the
query and a set of candidate documents respectively. Then, an ESIM model or
pre-trained language model, e.g. BERT or RoBERTa, is obtained as the scoring
function. Finally, the loss functions for the ranking task can be either
pair-wise or list-wise for training. The experimental results on the ART
dataset reach the state-of-the-art in the public leaderboard.
Related papers
- Graph Stochastic Neural Process for Inductive Few-shot Knowledge Graph Completion [63.68647582680998]
We focus on a task called inductive few-shot knowledge graph completion (I-FKGC)
Inspired by the idea of inductive reasoning, we cast I-FKGC as an inductive reasoning problem.
We present a neural process-based hypothesis extractor that models the joint distribution of hypothesis, from which we can sample a hypothesis for predictions.
In the second module, based on the hypothesis, we propose a graph attention-based predictor to test if the triple in the query set aligns with the extracted hypothesis.
arXiv Detail & Related papers (2024-08-03T13:37:40Z) - Ranking Large Language Models without Ground Truth [24.751931637152524]
Evaluation and ranking of large language models (LLMs) has become an important problem with the proliferation of these models.
We provide a novel perspective where, given a dataset of prompts, we rank them without access to any ground truth or reference responses.
Applying this idea repeatedly, we propose two methods to rank LLMs.
arXiv Detail & Related papers (2024-02-21T00:49:43Z) - Source-Free Unsupervised Domain Adaptation with Hypothesis Consolidation
of Prediction Rationale [53.152460508207184]
Source-Free Unsupervised Domain Adaptation (SFUDA) is a challenging task where a model needs to be adapted to a new domain without access to target domain labels or source domain data.
This paper proposes a novel approach that considers multiple prediction hypotheses for each sample and investigates the rationale behind each hypothesis.
To achieve the optimal performance, we propose a three-step adaptation process: model pre-adaptation, hypothesis consolidation, and semi-supervised learning.
arXiv Detail & Related papers (2024-02-02T05:53:22Z) - Large Language Models for Automated Open-domain Scientific Hypotheses Discovery [50.40483334131271]
This work proposes the first dataset for social science academic hypotheses discovery.
Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity.
A multi- module framework is developed for the task, including three different feedback mechanisms to boost performance.
arXiv Detail & Related papers (2023-09-06T05:19:41Z) - Logical Satisfiability of Counterfactuals for Faithful Explanations in
NLI [60.142926537264714]
We introduce the methodology of Faithfulness-through-Counterfactuals.
It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation.
It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
arXiv Detail & Related papers (2022-05-25T03:40:59Z) - Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text
Correspondence [45.9949173746044]
We show that large-size pre-trained language models (PLMs) do not satisfy the logical negation property (LNP)
We propose a novel intermediate training task, names meaning-matching, designed to directly learn a meaning-text correspondence.
We find that the task enables PLMs to learn lexical semantic information.
arXiv Detail & Related papers (2022-05-08T08:37:36Z) - Interactive Model with Structural Loss for Language-based Abductive
Reasoning [36.02450824915494]
The abductive natural language inference task ($alpha$NLI) is proposed to infer the most plausible explanation between the cause and the event.
We name this new model for $alpha$NLI: Interactive Model with Structural Loss (IMSL)
Our IMSL has achieved the highest performance on the RoBERTa-large pretrained model, with ACC and AUC results increased by about 1% and 5% respectively.
arXiv Detail & Related papers (2021-12-01T05:21:07Z) - Exact Distribution-Free Hypothesis Tests for the Regression Function of
Binary Classification via Conditional Kernel Mean Embeddings [0.0]
Two hypothesis tests are proposed for the regression function of binary classification based on conditional kernel mean embeddings.
Tests are introduced in a flexible manner allowing us to control the exact probability of type I error.
arXiv Detail & Related papers (2021-03-08T22:31:23Z) - Pre-training Is (Almost) All You Need: An Application to Commonsense
Reasoning [61.32992639292889]
Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks.
We introduce a new scoring method that casts a plausibility ranking task in a full-text format.
We show that our method provides a much more stable training phase across random restarts.
arXiv Detail & Related papers (2020-04-29T10:54:40Z) - HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in
Natural Language Inference [38.14399396661415]
We derive adversarial examples in terms of the hypothesis-only bias.
We investigate two debiasing approaches which exploit the artificial pattern modeling to mitigate such hypothesis-only bias.
arXiv Detail & Related papers (2020-03-05T16:46:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.