deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks

• URL: http://arxiv.org/abs/2204.06815v1
• Date: Thu, 14 Apr 2022 08:24:37 GMT
• Title: deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks
• Authors: Dennis Ulmer, Christian Hardmeier, Jes Frellsen
• Abstract summary: We provide an easy-to-use package containing different significance tests and utility functions. This package is specifically tailored towards research needs and usability.
• Score: 7.3372471678239215
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A lot of Machine Learning (ML) and Deep Learning (DL) research is of an empirical nature. Nevertheless, statistical significance testing (SST) is still not widely used. This endangers true progress, as seeming improvements over a baseline might be statistical flukes, leading follow-up research astray while wasting human and computational resources. Here, we provide an easy-to-use package containing different significance tests and utility functions specifically tailored towards research needs and usability.

Related papers

• Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks [100.13335639780415]
  Scaling laws often follow the power-law and proposed several variants of power-law functions to predict the scaling behavior at larger scales.<n>Existing methods mostly rely on point estimation and do not quantify uncertainty, which is crucial for real-world applications.<n>In this work, we explore a Bayesian framework based on Prior-data Fitted Networks (PFNs) for neural scaling law extrapolation.
  arXiv  Detail & Related papers  (2025-05-29T03:19:17Z)
• Meta-Statistical Learning: Supervised Learning of Statistical Inference [59.463430294611626]
  This work demonstrates that the tools and principles driving the success of large language models (LLMs) can be repurposed to tackle distribution-level tasks. We propose meta-statistical learning, a framework inspired by multi-instance learning that reformulates statistical inference tasks as supervised learning problems.
  arXiv  Detail & Related papers  (2025-02-17T18:04:39Z)
• Learning Representations for Independence Testing [13.842061060076004]
  We show how to construct powerful tests with finite-sample validity using variational estimators of mutual information.<n>Second, we establish a close connection between these variational mutual information-based tests and tests based on the Hilbert-Schmidt Independence Criterion (HSIC)<n>Finally, we show how to, rather than selecting a representation to maximize the statistic itself, select a representation which can maximize the power of a test.
  arXiv  Detail & Related papers  (2024-09-10T22:18:07Z)
• Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples [53.95282502030541]
  Neural Network-based active learning (NAL) is a cost-effective data selection technique that utilizes neural networks to select and train on a small subset of samples. We try to move one step forward by offering a unified explanation for the success of both query criteria-based NAL from a feature learning view.
  arXiv  Detail & Related papers  (2024-06-06T10:38:01Z)
• Task-Agnostic Machine-Learning-Assisted Inference [0.0]
  We introduce a novel statistical framework named PSPS for task-agnostic ML-assisted inference. PSPS provides a post-prediction inference solution that can be easily plugged into almost any established data analysis routines.
  arXiv  Detail & Related papers  (2024-05-30T13:19:49Z)
• MLXP: A Framework for Conducting Replicable Experiments in Python [63.37350735954699]
  We propose MLXP, an open-source, simple, and lightweight experiment management tool based on Python. It streamlines the experimental process with minimal overhead while ensuring a high level of practitioner overhead.
  arXiv  Detail & Related papers  (2024-02-21T14:22:20Z)
• Deep Learning for Multivariate Time Series Imputation: A Survey [34.31414745076129]
  Missing values are ubiquitous in multivariate time series (MTS) data.<n>Deep learning-based methods have successfully handled missing data by leveraging complex temporal dependencies and learned data distributions.<n>We propose a novel taxonomy that categorizes existing methods based on two key perspectives: imputation uncertainty and neural network architecture.
  arXiv  Detail & Related papers  (2024-02-06T15:03:53Z)
• Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck [35.6883212537938]
  We consider offline sparse parity learning, a supervised classification problem which admits a statistical query lower bound for gradient-based training of a multilayer perceptron. We show, theoretically and experimentally, that sparse initialization and increasing network width yield significant improvements in sample efficiency in this setting. We also show that the synthetic sparse parity task can be useful as a proxy for real problems requiring axis-aligned feature learning.
  arXiv  Detail & Related papers  (2023-09-07T15:52:48Z)
• Hypothesis Transfer Learning with Surrogate Classification Losses: Generalization Bounds through Algorithmic Stability [3.908842679355255]
  Hypothesis transfer learning (HTL) contrasts domain adaptation by allowing for a previous task leverage, named the source, into a new one, the target. This paper studies the learning theory of HTL through algorithmic stability, an attractive theoretical framework for machine learning algorithms analysis.
  arXiv  Detail & Related papers  (2023-05-31T09:38:21Z)
• A Sieve Quasi-likelihood Ratio Test for Neural Networks with Applications to Genetic Association Studies [0.32771631221674324]
  We propose a quasi sieve-likelihood ratio test based on NN with one hidden layer for testing complex associations. The validity of the distribution is investigated via simulations. We demonstrate the use of the proposed test by performing a genetic association analysis of the sequencing data from Alzheimer's Disease Neuroimaging Initiative (ADNI)
  arXiv  Detail & Related papers  (2022-12-16T02:54:46Z)
• Sample-Efficient Reinforcement Learning in the Presence of Exogenous Information [77.19830787312743]
  In real-world reinforcement learning applications the learner's observation space is ubiquitously high-dimensional with both relevant and irrelevant information about the task at hand. We introduce a new problem setting for reinforcement learning, the Exogenous Decision Process (ExoMDP), in which the state space admits an (unknown) factorization into a small controllable component and a large irrelevant component. We provide a new algorithm, ExoRL, which learns a near-optimal policy with sample complexity in the size of the endogenous component.
  arXiv  Detail & Related papers  (2022-06-09T05:19:32Z)
• Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study) [4.987581730476023]
  Test Input Prioritizers (TIP) for Deep Neural Networks (DNN) are an important technique to handle the typically very large test datasets efficiently. Feng et. al. propose DeepGini, a very fast and simple TIP, and show that it outperforms more elaborate techniques such as neuron- and surprise coverage.
  arXiv  Detail & Related papers  (2022-05-02T05:47:34Z)
• What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks? [59.51457877578138]
  We study contrastive learning on the wearable-based activity recognition task. This paper presents an open-source PyTorch library textttCL-HAR, which can serve as a practical tool for researchers.
  arXiv  Detail & Related papers  (2022-02-12T06:10:15Z)
• Deep neural networks approach to microbial colony detection -- a comparative analysis [52.77024349608834]
  This study investigates the performance of three deep learning approaches for object detection on the AGAR dataset. The achieved results may serve as a benchmark for future experiments.
  arXiv  Detail & Related papers  (2021-08-23T12:06:00Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.