Towards IID representation learning and its application on biomedical data

• URL: http://arxiv.org/abs/2203.00332v1
• Date: Tue, 1 Mar 2022 10:15:14 GMT
• Title: Towards IID representation learning and its application on biomedical data
• Abstract summary: The widely accepted independent and identically distributed (IID) assumption has been criticized in recent studies on causality. In this paper, we argue that instead of being a questionable assumption, IID is a fundamental task-relevant property that needs to be learned.
• Score: 2.91364210254152
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Due to the heterogeneity of real-world data, the widely accepted independent and identically distributed (IID) assumption has been criticized in recent studies on causality. In this paper, we argue that instead of being a questionable assumption, IID is a fundamental task-relevant property that needs to be learned. Consider $k$ independent random vectors $\mathsf{X}^{i = 1, \ldots, k}$, we elaborate on how a variety of different causal questions can be reformulated to learning a task-relevant function $\phi$ that induces IID among $\mathsf{Z}^i := \phi \circ \mathsf{X}^i$, which we term IID representation learning. For proof of concept, we examine the IID representation learning on Out-of-Distribution (OOD) generalization tasks. Concretely, by utilizing the representation obtained via the learned function that induces IID, we conduct prediction of molecular characteristics (molecular prediction) on two biomedical datasets with real-world distribution shifts introduced by a) preanalytical variation and b) sampling protocol. To enable reproducibility and for comparison to the state-of-the-art (SOTA) methods, this is done by following the OOD benchmarking guidelines recommended from WILDS. Compared to the SOTA baselines supported in WILDS, the results confirm the superior performance of IID representation learning on OOD tasks. The code is publicly accessible via https://github.com/CTPLab/IID_representation_learning.

Related papers

• Does Your Neural Network Extrapolate? Feature Engineering as Identifiability Bias for OOD Generalization [1.0989593035411862]
  Successful deep neural networks discover salient features of data.<n>We show when and why they fail to learn out-of-distribution (OOD)-relevant representations from an in-distribution (ID) training window.
  arXiv  Detail & Related papers  (2026-05-08T09:25:52Z)
• Pairwise Matching of Intermediate Representations for Fine-grained Explainability [7.415710605852485]
  We propose a new explainability method (PAIR-X) to generate fine-grained, highly-localized pairwise visual explanations. By improving interpretability, PAIR-X enables humans to better distinguish correct and incorrect matches.
  arXiv  Detail & Related papers  (2025-03-28T21:13:43Z)
• FedUV: Uniformity and Variance for Heterogeneous Federated Learning [5.9330433627374815]
  Federated learning is a promising framework to train neural networks with widely distributed data. Recent work has shown this is due to the final layer of the network being most prone to local bias. We investigate the training dynamics of the classifier by applying SVD to the weights motivated by the observation that freezing weights results in constant singular values.
  arXiv  Detail & Related papers  (2024-02-27T15:53:15Z)
• Collaborative Learning with Different Labeling Functions [7.228285747845779]
  We study a variant of Collaborative PAC Learning, in which we aim to learn an accurate classifier for each of the $n$ data distributions. We show that, when the data distributions satisfy a weaker realizability assumption, sample-efficient learning is still feasible.
  arXiv  Detail & Related papers  (2024-02-16T04:32:22Z)
• How Does Unlabeled Data Provably Help Out-of-Distribution Detection? [63.41681272937562]
  Unlabeled in-the-wild data is non-trivial due to the heterogeneity of both in-distribution (ID) and out-of-distribution (OOD) data. This paper introduces a new learning framework SAL (Separate And Learn) that offers both strong theoretical guarantees and empirical effectiveness.
  arXiv  Detail & Related papers  (2024-02-05T20:36:33Z)
• Learning From Drift: Federated Learning on Non-IID Data via Drift Regularization [11.813552364878868]
  Federated learning algorithms perform reasonably well on independent and identically distributed (IID) data. They suffer greatly from heterogeneous environments, i.e., Non-IID data. We propose Learning from Drift (LfD), a novel method for effectively training the model in heterogeneous settings.
  arXiv  Detail & Related papers  (2023-09-13T09:23:09Z)
• DCID: Deep Canonical Information Decomposition [84.59396326810085]
  We consider the problem of identifying the signal shared between two one-dimensional target variables. We propose ICM, an evaluation metric which can be used in the presence of ground-truth labels. We also propose Deep Canonical Information Decomposition (DCID) - a simple, yet effective approach for learning the shared variables.
  arXiv  Detail & Related papers  (2023-06-27T16:59:06Z)
• Open-World Pose Transfer via Sequential Test-Time Adaption [92.67291699304992]
  A typical pose transfer framework usually employs representative datasets to train a discriminative model. Test-time adaption (TTA) offers a feasible solution for OOD data by using a pre-trained model that learns essential features with self-supervision. In our experiment, we first show that pose transfer can be applied to open-world applications, including Tiktok reenactment and celebrity motion synthesis.
  arXiv  Detail & Related papers  (2023-03-20T09:01:23Z)
• Do Deep Neural Networks Always Perform Better When Eating More Data? [82.6459747000664]
  We design experiments from Identically Independent Distribution(IID) and Out of Distribution(OOD) Under IID condition, the amount of information determines the effectivity of each sample, the contribution of samples and difference between classes determine the amount of class information. Under OOD condition, the cross-domain degree of samples determine the contributions, and the bias-fitting caused by irrelevant elements is a significant factor of cross-domain.
  arXiv  Detail & Related papers  (2022-05-30T15:40:33Z)
• No Shifted Augmentations (NSA): compact distributions for robust self-supervised Anomaly Detection [4.243926243206826]
  Unsupervised Anomaly detection (AD) requires building a notion of normalcy, distinguishing in-distribution (ID) and out-of-distribution (OOD) data. We investigate how the emph geometrical compactness of the ID feature distribution makes isolating and detecting outliers easier. We propose novel architectural modifications to the self-supervised feature learning step, that enable such compact distributions for ID data to be learned.
  arXiv  Detail & Related papers  (2022-03-19T15:55:32Z)
• Covariate-informed Representation Learning with Samplewise Optimal Identifiable Variational Autoencoders [15.254297587065595]
  Recently proposed identifiable variational autoencoder (iVAE) provides a promising approach for learning latent independent components of the data. We develop a new approach, co-informed identifiable VAE (CI-iVAE) In doing so, the objective function enforces the inverse relation, and learned representation contains more information of observations.
  arXiv  Detail & Related papers  (2022-02-09T00:18:33Z)
• Introspective Distillation for Robust Question Answering [70.18644911309468]
  Question answering (QA) models are well-known to exploit data bias, e.g., the language prior in visual QA and the position bias in reading comprehension. Recent debiasing methods achieve good out-of-distribution (OOD) generalizability with a considerable sacrifice of the in-distribution (ID) performance. We present a novel debiasing method called Introspective Distillation (IntroD) to make the best of both worlds for QA.
  arXiv  Detail & Related papers  (2021-11-01T15:30:15Z)
• IID-GAN: an IID Sampling Perspective for Regularizing Mode Collapse [82.49564071049366]
  generative adversarial networks (GANs) still suffer from mode collapse. We analyze and seek to regularize this issue with an independent and identically distributed (IID) sampling perspective. We propose a new loss to encourage the closeness between inverse samples of real data and the Gaussian source in latent space to regularize the generation to be IID from the target distribution.
  arXiv  Detail & Related papers  (2021-06-01T15:20:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.