Calibrating dimension reduction hyperparameters in the presence of noise

• URL: http://arxiv.org/abs/2312.02946v5
• Date: Thu, 11 Jul 2024 19:27:19 GMT
• Title: Calibrating dimension reduction hyperparameters in the presence of noise
• Authors: Justin Lin, Julia Fukuyama,
• Abstract summary: t-SNE and UMAP fail to acknowledge data as a combination of signal and noise when assessing performance. We show previously recommended values for perplexity and n_neighbors are too small and overfit the noise.
• Score: 0.4143603294943439
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The goal of dimension reduction tools is to construct a low-dimensional representation of high-dimensional data. These tools are employed for a variety of reasons such as noise reduction, visualization, and to lower computational costs. However, there is a fundamental issue that is discussed in other modeling problems that is often overlooked in dimension reduction -- overfitting. In the context of other modeling problems, techniques such as feature-selection, cross-validation, and regularization are employed to combat overfitting, but rarely are such precautions taken when applying dimension reduction. Prior applications of the two most popular non-linear dimension reduction methods, t-SNE and UMAP, fail to acknowledge data as a combination of signal and noise when assessing performance. These methods are typically calibrated to capture the entirety of the data, not just the signal. In this paper, we demonstrate the importance of acknowledging noise when calibrating hyperparameters and present a framework that enables users to do so. We use this framework to explore the role hyperparameter calibration plays in overfitting the data when applying t-SNE and UMAP. More specifically, we show previously recommended values for perplexity and n_neighbors are too small and overfit the noise. We also provide a workflow others may use to calibrate hyperparameters in the presence of noise.

Related papers

• One-step Noisy Label Mitigation [86.57572253460125]
  Mitigating the detrimental effects of noisy labels on the training process has become increasingly critical. We propose One-step Anti-Noise (OSA), a model-agnostic noisy label mitigation paradigm. We empirically demonstrate the superiority of OSA, highlighting its enhanced training robustness, improved task transferability, ease of deployment, and reduced computational costs.
  arXiv  Detail & Related papers  (2024-10-02T18:42:56Z)
• SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Models [85.67096251281191]
  We present an innovative approach to model fusion called zero-shot Sparse MIxture of Low-rank Experts (SMILE) construction. SMILE allows for the upscaling of source models into an MoE model without extra data or further training. We conduct extensive experiments across diverse scenarios, such as image classification and text generation tasks, using full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2024-08-19T17:32:15Z)
• Robust shallow shadows [0.251657752676152]
  We present a robust shadow estimation protocol for wide classes of shallow measurement circuits. We show how to estimate this directly from experimental data using tensor-network tools. Under the practical constraints of current and near-term noisy quantum devices, our method maximally realizes the potential of shadow estimation with global rotations.
  arXiv  Detail & Related papers  (2024-05-09T18:00:09Z)
• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Label Noise: Correcting the Forward-Correction [0.0]
  Training neural network classifiers on datasets with label noise poses a risk of overfitting them to the noisy labels. We propose an approach to tackling overfitting caused by label noise. Motivated by this observation, we propose imposing a lower bound on the training loss to mitigate overfitting.
  arXiv  Detail & Related papers  (2023-07-24T19:41:19Z)
• Improve Noise Tolerance of Robust Loss via Noise-Awareness [60.34670515595074]
  We propose a meta-learning method which is capable of adaptively learning a hyper parameter prediction function, called Noise-Aware-Robust-Loss-Adjuster (NARL-Adjuster for brevity) Four SOTA robust loss functions are attempted to be integrated with our algorithm, and comprehensive experiments substantiate the general availability and effectiveness of the proposed method in both its noise tolerance and performance.
  arXiv  Detail & Related papers  (2023-01-18T04:54:58Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)
• Bayesian NVH metamodels to assess interior cabin noise using measurement databases [0.0]
  This research work proposes a global NVH metamodeling technique for broadband noises such as aerodynamic and rolling noises. Generalized additive models (GAMs) with bootstraps and Gaussian basis functions are used to model the dependency of sound pressure level (SPL) on predictor variables. Probabilistic modelling is carried out using an open-source library PyMC3.
  arXiv  Detail & Related papers  (2022-06-12T19:48:24Z)
• Hierarchical model reduction driven by machine learning for parametric advection-diffusion-reaction problems in the presence of noisy data [0.0]
  We propose a new approach to generate a reliable reduced model for a parametric elliptic problem in the presence of noisy data. We show that directional HiPOD looses in terms of accuracy when problem data are affected by noise. We replace with Machine Learning fitting models which better discriminate relevant physical features in the data from irrelevant noise.
  arXiv  Detail & Related papers  (2022-04-01T16:02:05Z)
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance [76.54300276636982]
  Noise in gradient descent provides a crucial implicit regularization effect for training over parameterized models. We show that parameter-dependent noise -- induced by mini-batches or label perturbation -- is far more effective than Gaussian noise. Our analysis reveals that parameter-dependent noise introduces a bias towards local minima with smaller noise variance, whereas spherical Gaussian noise does not.
  arXiv  Detail & Related papers  (2020-06-15T18:31:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.