Continual Feature Selection: Spurious Features in Continual Learning
- URL: http://arxiv.org/abs/2203.01012v1
- Date: Wed, 2 Mar 2022 10:43:54 GMT
- Title: Continual Feature Selection: Spurious Features in Continual Learning
- Authors: Timoth\'ee Lesort
- Abstract summary: This paper studies spurious features' influence on continual learning algorithms.
We show that learning algorithms solve tasks by overfitting features that are not generalizable.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Continual Learning (CL) is the research field addressing learning settings
where the data distribution is not static. This paper studies spurious
features' influence on continual learning algorithms. Indeed, we show that
learning algorithms solve tasks by overfitting features that are not
generalizable. To better understand these phenomena and their impact, we
propose a domain incremental scenario that we study through various
out-of-distribution generalizations and continual learning algorithms. The
experiments of this paper show that continual learning algorithms face two
related challenges: (1) the spurious features challenge: some features are well
correlated with labels in train data but not in test data due to a covariate
shift between train and test. (2) the local spurious features challenge: some
features correlate well with labels within a task but not within the whole task
sequence. The challenge is to learn general features that are neither spurious
(in general) nor locally spurious. We prove that the latter is a major cause of
performance decrease in continual learning along with catastrophic forgetting.
Our results indicate that the best solution to overcome the feature selection
problems varies depending on the correlation between spurious features (SFs)
and labels. The vanilla replay approach seems to be a powerful approach to deal
with SFs, which could explain its good performance in the continual learning
literature. This paper presents a different way of understanding performance
decrease in continual learning by describing the influence of spurious/local
spurious features.
Related papers
- Complexity Matters: Dynamics of Feature Learning in the Presence of Spurious Correlations [13.119576365743624]
We study the dynamics of feature learning under spurious correlations.
We demonstrate that our findings justify the success of retraining the last layer to remove spurious correlation.
We also identify limitations of popular debiasing algorithms that exploit early learning of spurious features.
arXiv Detail & Related papers (2024-03-05T23:54:00Z) - Causal Feature Selection via Transfer Entropy [59.999594949050596]
Causal discovery aims to identify causal relationships between features with observational data.
We introduce a new causal feature selection approach that relies on the forward and backward feature selection procedures.
We provide theoretical guarantees on the regression and classification errors for both the exact and the finite-sample cases.
arXiv Detail & Related papers (2023-10-17T08:04:45Z) - Long-Tailed Recognition by Mutual Information Maximization between
Latent Features and Ground-Truth Labels [10.782043595405831]
This paper integrates contrastive learning and logit adjustment to derive a loss function that shows state-of-the-art performance on longtailed recognition benchmarks.
It also demonstrates its efficacy in image segmentation tasks, verifying its imbalances beyond image classification.
arXiv Detail & Related papers (2023-05-02T02:29:18Z) - Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting [34.402943976801424]
We study the coexistence of two phenomena that affect the quality of continually learned representations: knowledge accumulation and feature forgetting.
We show that even though forgetting in the representation can be small in absolute terms, forgetting in the representation tends to be just as catastrophic as forgetting at the output level.
arXiv Detail & Related papers (2023-04-03T12:45:52Z) - Multi-task Bias-Variance Trade-off Through Functional Constraints [102.64082402388192]
Multi-task learning aims to acquire a set of functions that perform well for diverse tasks.
In this paper we draw intuition from the two extreme learning scenarios -- a single function for all tasks, and a task-specific function that ignores the other tasks.
We introduce a constrained learning formulation that enforces domain specific solutions to a central function.
arXiv Detail & Related papers (2022-10-27T16:06:47Z) - ALLSH: Active Learning Guided by Local Sensitivity and Hardness [98.61023158378407]
We propose to retrieve unlabeled samples with a local sensitivity and hardness-aware acquisition function.
Our method achieves consistent gains over the commonly used active learning strategies in various classification tasks.
arXiv Detail & Related papers (2022-05-10T15:39:11Z) - Automated Self-Supervised Learning for Graphs [37.14382990139527]
This work aims to investigate how to automatically leverage multiple pretext tasks effectively.
We make use of a key principle of many real-world graphs, i.e., homophily, as the guidance to effectively search various self-supervised pretext tasks.
We propose the AutoSSL framework which can automatically search over combinations of various self-supervised tasks.
arXiv Detail & Related papers (2021-06-10T03:09:20Z) - Distribution Matching for Heterogeneous Multi-Task Learning: a
Large-scale Face Study [75.42182503265056]
Multi-Task Learning has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm.
We deal with heterogeneous MTL, simultaneously addressing detection, classification & regression problems.
We build FaceBehaviorNet, the first framework for large-scale face analysis, by jointly learning all facial behavior tasks.
arXiv Detail & Related papers (2021-05-08T22:26:52Z) - Multi-task Supervised Learning via Cross-learning [102.64082402388192]
We consider a problem known as multi-task learning, consisting of fitting a set of regression functions intended for solving different tasks.
In our novel formulation, we couple the parameters of these functions, so that they learn in their task specific domains while staying close to each other.
This facilitates cross-fertilization in which data collected across different domains help improving the learning performance at each other task.
arXiv Detail & Related papers (2020-10-24T21:35:57Z) - Learning Causal Models Online [103.87959747047158]
Predictive models can rely on spurious correlations in the data for making predictions.
One solution for achieving strong generalization is to incorporate causal structures in the models.
We propose an online algorithm that continually detects and removes spurious features.
arXiv Detail & Related papers (2020-06-12T20:49:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.