Interpretable Research Replication Prediction via Variational Contextual Consistency Sentence Masking

• URL: http://arxiv.org/abs/2203.14474v1
• Date: Mon, 28 Mar 2022 03:27:13 GMT
• Title: Interpretable Research Replication Prediction via Variational Contextual Consistency Sentence Masking
• Authors: Tianyi Luo, Rui Meng, Xin Eric Wang, Yang Liu
• Abstract summary: Research Replication Prediction (RRP) is the task of predicting whether a published research result can be replicated or not. In this work, we propose the Variational Contextual Consistency Sentence Masking (VCCSM) method to automatically extract key sentences. Results of our experiments on RRP along with European Convention of Human Rights (ECHR) datasets demonstrate that VCCSM is able to improve the model interpretability for the long document classification tasks.
• Score: 14.50690911709558
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Research Replication Prediction (RRP) is the task of predicting whether a published research result can be replicated or not. Building an interpretable neural text classifier for RRP promotes the understanding of why a research paper is predicted as replicable or non-replicable and therefore makes its real-world application more reliable and trustworthy. However, the prior works on model interpretation mainly focused on improving the model interpretability at the word/phrase level, which are insufficient especially for long research papers in RRP. Furthermore, the existing methods cannot utilize a large size of unlabeled dataset to further improve the model interpretability. To address these limitations, we aim to build an interpretable neural model which can provide sentence-level explanations and apply weakly supervised approach to further leverage the large corpus of unlabeled datasets to boost the interpretability in addition to improving prediction performance as existing works have done. In this work, we propose the Variational Contextual Consistency Sentence Masking (VCCSM) method to automatically extract key sentences based on the context in the classifier, using both labeled and unlabeled datasets. Results of our experiments on RRP along with European Convention of Human Rights (ECHR) datasets demonstrate that VCCSM is able to improve the model interpretability for the long document classification tasks using the area over the perturbation curve and post-hoc accuracy as evaluation metrics.

Related papers

• Causal Rule Forest: Toward Interpretable and Precise Treatment Effect Estimation [0.0]
  Causal Rule Forest (CRF) is a novel approach to learning hidden patterns from data and transforming the patterns into interpretable multi-level Boolean rules. By training the other interpretable causal inference models with data representation learned by CRF, we can reduce the predictive errors of these models in estimating Heterogeneous Treatment Effects (HTE) and Conditional Average Treatment Effects (CATE) Our experiments underscore the potential of CRF to advance personalized interventions and policies.
  arXiv  Detail & Related papers  (2024-08-27T13:32:31Z)
• Ranking and Combining Latent Structured Predictive Scores without Labeled Data [2.5064967708371553]
  This paper introduces a novel structured unsupervised ensemble learning model (SUEL) It exploits the dependency between a set of predictors with continuous predictive scores, rank the predictors without labeled data and combine them to an ensembled score with weights. The efficacy of the proposed methods is rigorously assessed through both simulation studies and real-world application of risk genes discovery.
  arXiv  Detail & Related papers  (2024-08-14T20:14:42Z)
• Enhancing Retrieval-Augmented LMs with a Two-stage Consistency Learning Compressor [4.35807211471107]
  This work proposes a novel two-stage consistency learning approach for retrieved information compression in retrieval-augmented language models. The proposed method is empirically validated across multiple datasets, demonstrating notable enhancements in precision and efficiency for question-answering tasks.
  arXiv  Detail & Related papers  (2024-06-04T12:43:23Z)
• RDR: the Recap, Deliberate, and Respond Method for Enhanced Language Understanding [6.738409533239947]
  The Recap, Deliberate, and Respond (RDR) paradigm addresses this issue by incorporating three distinct objectives within the neural network pipeline. By cascading these three models, we mitigate the potential for gaming the benchmark and establish a robust method for capturing the underlying semantic patterns. Our results demonstrate improved performance compared to competitive baselines, with an enhancement of up to 2% on standard metrics.
  arXiv  Detail & Related papers  (2023-12-15T16:41:48Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• A comprehensive comparative evaluation and analysis of Distributional Semantic Models [61.41800660636555]
  We perform a comprehensive evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. The results show that the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous. We borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models.
  arXiv  Detail & Related papers  (2021-05-20T15:18:06Z)
• Generative Counterfactuals for Neural Networks via Attribute-Informed Perturbation [51.29486247405601]
  We design a framework to generate counterfactuals for raw data instances with the proposed Attribute-Informed Perturbation (AIP) By utilizing generative models conditioned with different attributes, counterfactuals with desired labels can be obtained effectively and efficiently. Experimental results on real-world texts and images demonstrate the effectiveness, sample quality as well as efficiency of our designed framework.
  arXiv  Detail & Related papers  (2021-01-18T08:37:13Z)
• An Investigation of Language Model Interpretability via Sentence Editing [5.492504126672887]
  We re-purpose a sentence editing dataset as a testbed for interpretability of pre-trained language models (PLMs) This enables us to conduct a systematic investigation on an array of questions regarding PLMs' interpretability. The investigation generates new insights, for example, contrary to the common understanding, we find that attention weights correlate well with human rationales.
  arXiv  Detail & Related papers  (2020-11-28T00:46:43Z)
• Open-set Short Utterance Forensic Speaker Verification using Teacher-Student Network with Explicit Inductive Bias [59.788358876316295]
  We propose a pipeline solution to improve speaker verification on a small actual forensic field dataset. By leveraging large-scale out-of-domain datasets, a knowledge distillation based objective function is proposed for teacher-student learning. We show that the proposed objective function can efficiently improve the performance of teacher-student learning on short utterances.
  arXiv  Detail & Related papers  (2020-09-21T00:58:40Z)
• ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning [85.33459673197149]
  We introduce a new Reading dataset requiring logical reasoning (ReClor) extracted from standardized graduate admission examinations. In this paper, we propose to identify biased data points and separate them into EASY set and the rest as HARD set. Empirical results show that state-of-the-art models have an outstanding ability to capture biases contained in the dataset with high accuracy on EASY set. However, they struggle on HARD set with poor performance near that of random guess, indicating more research is needed to essentially enhance the logical reasoning ability of current models.
  arXiv  Detail & Related papers  (2020-02-11T11:54:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.