The Impact of Edge Displacement Vaserstein Distance on UD Parsing Performance

• URL: http://arxiv.org/abs/2209.07139v1
• Date: Thu, 15 Sep 2022 08:37:12 GMT
• Title: The Impact of Edge Displacement Vaserstein Distance on UD Parsing Performance
• Authors: Mark Anderson, Carlos G\'omez-Rodr\'iguez
• Abstract summary: We introduce a measurement that evaluates the differences between the distributions of edge displacement seen in training and test data. We then attempt to falsify this hypothesis by using a number of statistical methods. In a broader sense, the methodology presented here can act as a reference for future correlation-based exploratory work in NLP.
• Score: 3.7311680121118345
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We contribute to the discussion on parsing performance in NLP by introducing a measurement that evaluates the differences between the distributions of edge displacement (the directed distance of edges) seen in training and test data. We hypothesize that this measurement will be related to differences observed in parsing performance across treebanks. We motivate this by building upon previous work and then attempt to falsify this hypothesis by using a number of statistical methods. We establish that there is a statistical correlation between this measurement and parsing performance even when controlling for potential covariants. We then use this to establish a sampling technique that gives us an adversarial and complementary split. This gives an idea of the lower and upper bounds of parsing systems for a given treebank in lieu of freshly sampled data. In a broader sense, the methodology presented here can act as a reference for future correlation-based exploratory work in NLP.

Related papers

• Rethinking Distance Metrics for Counterfactual Explainability [53.436414009687]
  We investigate a framing for counterfactual generation methods that considers counterfactuals not as independent draws from a region around the reference, but as jointly sampled with the reference from the underlying data distribution. We derive a distance metric, tailored for counterfactual similarity that can be applied to a broad range of settings.
  arXiv  Detail & Related papers  (2024-10-18T15:06:50Z)
• Combine and Conquer: A Meta-Analysis on Data Shift and Out-of-Distribution Detection [30.377446496559635]
  This paper introduces a universal approach to seamlessly combine out-of-distribution (OOD) detection scores. Our framework is easily for future developments in detection scores and stands as the first to combine decision boundaries in this context.
  arXiv  Detail & Related papers  (2024-06-23T08:16:44Z)
• Efficient Epistemic Uncertainty Estimation in Regression Ensemble Models Using Pairwise-Distance Estimators [21.098866735156207]
  Pairwise-distance estimators (PaiDEs) establish bounds on entropy. Unlike sample-based Monte Carlo estimators, PaiDEs exhibit a remarkable capability to estimate epistemic uncertainty at speeds up to 100 times faster. We compare our approach to existing active learning methods and find that our approach outperforms on high-dimensional regression tasks.
  arXiv  Detail & Related papers  (2023-08-25T17:13:42Z)
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories [61.3299263929289]
  Simulation-based inference enables learning the parameters of a model even when its likelihood cannot be computed in practice. One class of methods uses data simulated with different parameters to infer an amortized estimator for the likelihood-to-evidence ratio. We show that this approach can be formulated in terms of mutual information between model parameters and simulated data.
  arXiv  Detail & Related papers  (2021-06-03T12:59:16Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• Efficient Data-Dependent Learnability [8.766022970635898]
  The predictive normalized maximum likelihood (pNML) approach has recently been proposed as the min-max optimal solution to the batch learning problem. We show that when applied to neural networks, this approximation can detect out-of-distribution examples effectively.
  arXiv  Detail & Related papers  (2020-11-20T10:44:55Z)
• Optimal Off-Policy Evaluation from Multiple Logging Policies [77.62012545592233]
  We study off-policy evaluation from multiple logging policies, each generating a dataset of fixed size, i.e., stratified sampling. We find the OPE estimator for multiple loggers with minimum variance for any instance, i.e., the efficient one.
  arXiv  Detail & Related papers  (2020-10-21T13:43:48Z)
• Almost-Matching-Exactly for Treatment Effect Estimation under Network Interference [73.23326654892963]
  We propose a matching method that recovers direct treatment effects from randomized experiments where units are connected in an observed network. Our method matches units almost exactly on counts of unique subgraphs within their neighborhood graphs.
  arXiv  Detail & Related papers  (2020-03-02T15:21:20Z)
• An end-to-end approach for the verification problem: learning the right distance [15.553424028461885]
  We augment the metric learning setting by introducing a parametric pseudo-distance, trained jointly with the encoder. We first show it approximates a likelihood ratio which can be used for hypothesis tests. We observe training is much simplified under the proposed approach compared to metric learning with actual distances.
  arXiv  Detail & Related papers  (2020-02-21T18:46:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.