Structurally Diverse Sampling Reduces Spurious Correlations in Semantic
Parsing Datasets
- URL: http://arxiv.org/abs/2203.08445v1
- Date: Wed, 16 Mar 2022 07:41:27 GMT
- Title: Structurally Diverse Sampling Reduces Spurious Correlations in Semantic
Parsing Datasets
- Authors: Shivanshu Gupta and Sameer Singh and Matt Gardner
- Abstract summary: We propose a novel algorithm for sampling a structurally diverse set of instances from a labeled instance pool with structured outputs.
We show that our algorithm performs competitively with or better than prior algorithms in not only compositional template splits but also traditional IID splits.
In general, we find that diverse train sets lead to better generalization than random training sets of the same size in 9 out of 10 dataset-split pairs.
- Score: 51.095144091781734
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: A rapidly growing body of research has demonstrated the inability of NLP
models to generalize compositionally and has tried to alleviate it through
specialized architectures, training schemes, and data augmentation, among other
approaches. In this work, we study a different relatively under-explored
approach: sampling diverse train sets that encourage compositional
generalization. We propose a novel algorithm for sampling a structurally
diverse set of instances from a labeled instance pool with structured outputs.
Evaluating on 5 semantic parsing datasets of varying complexity, we show that
our algorithm performs competitively with or better than prior algorithms in
not only compositional template splits but also traditional IID splits of all
but the least structurally diverse datasets. In general, we find that diverse
train sets lead to better generalization than random training sets of the same
size in 9 out of 10 dataset-split pairs, with over 10% absolute improvement in
5, providing further evidence to their sample efficiency. Moreover, we show
that structural diversity also makes for more comprehensive test sets that
require diverse training to succeed on. Finally, we use information theory to
show that reduction in spurious correlations between substructures may be one
reason why diverse training sets improve generalization.
Related papers
- Learning Dynamic Bayesian Networks from Data: Foundations, First Principles and Numerical Comparisons [2.403231673869682]
We present a guide to the foundations of learning Dynamic Bayesian Networks (DBNs) from data.
We present the formalism for a generic as well as a set of common types of DBNs for particular variable distributions.
arXiv Detail & Related papers (2024-06-25T14:28:17Z) - ComboStoc: Combinatorial Stochasticity for Diffusion Generative Models [65.82630283336051]
We show that the space spanned by the combination of dimensions and attributes is insufficiently sampled by existing training scheme of diffusion generative models.
We present a simple fix to this problem by constructing processes that fully exploit the structures, hence the name ComboStoc.
arXiv Detail & Related papers (2024-05-22T15:23:10Z) - Deep Negative Correlation Classification [82.45045814842595]
Existing deep ensemble methods naively train many different models and then aggregate their predictions.
We propose deep negative correlation classification (DNCC)
DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
arXiv Detail & Related papers (2022-12-14T07:35:20Z) - Learning to Generalize Compositionally by Transferring Across Semantic
Parsing Tasks [37.66114618645146]
We investigate learning representations that facilitate transfer learning from one compositional task to another.
We apply this method to semantic parsing, using three very different datasets.
Our method significantly improves compositional generalization over baselines on the test set of the target task.
arXiv Detail & Related papers (2021-11-09T09:10:21Z) - Cross-Dataset Collaborative Learning for Semantic Segmentation [17.55660581677053]
We present a simple, flexible, and general method for semantic segmentation, termed Cross-Dataset Collaborative Learning (CDCL)
Given multiple labeled datasets, we aim to improve the generalization and discrimination of feature representations on each dataset.
We conduct extensive evaluations on four diverse datasets, i.e., Cityscapes, BDD100K, CamVid, and COCO Stuff, with single-dataset and cross-dataset settings.
arXiv Detail & Related papers (2021-03-21T09:59:47Z) - Substructure Substitution: Structured Data Augmentation for NLP [55.69800855705232]
SUB2 generates new examples by substituting substructures with ones with the same label.
For more general tasks, we present variations of SUB2 based on constituency parse trees.
For most cases, training with the augmented dataset by SUB2 achieves better performance than training with the original training set.
arXiv Detail & Related papers (2021-01-02T09:54:24Z) - A Comparative Analysis of the Ensemble Methods for Drug Design [0.0]
Ensemble-based machine learning approaches have been used to overcome limitations and generate reliable predictions.
In this article, 57 algorithms were developed and compared on 4 different datasets.
The proposed individual models did not show impressive results as a unified model, but it was considered the most important predictor when combined.
arXiv Detail & Related papers (2020-12-11T05:27:20Z) - CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural
Summarization Systems [121.78477833009671]
We investigate the performance of different summarization models under a cross-dataset setting.
A comprehensive study of 11 representative summarization systems on 5 datasets from different domains reveals the effect of model architectures and generation ways.
arXiv Detail & Related papers (2020-10-11T02:19:15Z) - Neural Ensemble Search for Uncertainty Estimation and Dataset Shift [67.57720300323928]
Ensembles of neural networks achieve superior performance compared to stand-alone networks in terms of accuracy, uncertainty calibration and robustness to dataset shift.
We propose two methods for automatically constructing ensembles with emphvarying architectures.
We show that the resulting ensembles outperform deep ensembles not only in terms of accuracy but also uncertainty calibration and robustness to dataset shift.
arXiv Detail & Related papers (2020-06-15T17:38:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.