Studying Catastrophic Forgetting in Neural Ranking Models

• URL: http://arxiv.org/abs/2101.06984v1
• Date: Mon, 18 Jan 2021 10:42:57 GMT
• Title: Studying Catastrophic Forgetting in Neural Ranking Models
• Authors: Jesus Lovon-Melgarejo, Laure Soulier, Karen Pinel-Sauvagnat, Lynda Tamine
• Abstract summary: We study in what extent neural ranking models catastrophically forget old knowledge acquired from previously observed domains after acquiring new knowledge. Our experiments show that the effectiveness of neuralIR ranking models is achieved at the cost of catastrophic forgetting. We believe that the obtained results can be useful for both theoretical and practical future work in neural IR.
• Score: 3.8596788671326947
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Several deep neural ranking models have been proposed in the recent IR literature. While their transferability to one target domain held by a dataset has been widely addressed using traditional domain adaptation strategies, the question of their cross-domain transferability is still under-studied. We study here in what extent neural ranking models catastrophically forget old knowledge acquired from previously observed domains after acquiring new knowledge, leading to performance decrease on those domains. Our experiments show that the effectiveness of neuralIR ranking models is achieved at the cost of catastrophic forgetting and that a lifelong learning strategy using a cross-domain regularizer success-fully mitigates the problem. Using an explanatory approach built on a regression model, we also show the effect of domain characteristics on the rise of catastrophic forgetting. We believe that the obtained results can be useful for both theoretical and practical future work in neural IR.

Related papers

• SEANN: A Domain-Informed Neural Network for Epidemiological Insights [0.9749638953163389]
  We introduce SEANN, a novel approach for informed agnostics that leverages a prevalent form of domain-specific knowledge: Pooled Effect Sizes (PES) PESs are commonly found in published Meta-Analysis studies, in different forms, and represent a quantitative form of a scientific consensus. We experimentally demonstrate significant improvements in the generalizability of predictive performances and the scientific plausibility of extracted relationships.
  arXiv  Detail & Related papers  (2025-01-17T16:01:05Z)
• Predicting Resilience with Neural Networks [1.4425878137951238]
  Resilience engineering studies the ability of a system to survive and recover from disruptive events. This paper proposes three alternative neural network (NN) approaches to model and predict system performance.
  arXiv  Detail & Related papers  (2023-08-11T17:29:49Z)
• Principles of Forgetting in Domain-Incremental Semantic Segmentation in Adverse Weather Conditions [0.0]
  Adverse weather conditions can significantly decrease model performance when such data are not available during training. We study how the representations of semantic segmentation models are affected during domain-incremental learning in adverse weather conditions. Our experiments and representational analyses indicate that catastrophic forgetting is primarily caused by changes to low-level features in domain-incremental learning.
  arXiv  Detail & Related papers  (2023-03-24T16:23:49Z)
• Forget Less, Count Better: A Domain-Incremental Self-Distillation Learning Benchmark for Lifelong Crowd Counting [51.44987756859706]
  Off-the-shelf methods have some drawbacks to handle multiple domains. Lifelong Crowd Counting aims at alleviating the catastrophic forgetting and improving the generalization ability.
  arXiv  Detail & Related papers  (2022-05-06T15:37:56Z)
• Algorithms and Theory for Supervised Gradual Domain Adaptation [19.42476993856205]
  We study the problem of supervised gradual domain adaptation, where labeled data from shifting distributions are available to the learner along the trajectory. Under this setting, we provide the first generalization upper bound on the learning error under mild assumptions. Our results are algorithm agnostic for a range of loss functions, and only depend linearly on the averaged learning error across the trajectory.
  arXiv  Detail & Related papers  (2022-04-25T13:26:11Z)
• On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
  Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
  arXiv  Detail & Related papers  (2022-03-08T09:57:48Z)
• Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay [67.50637511633212]
  A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
  arXiv  Detail & Related papers  (2021-12-09T07:11:14Z)
• f-Domain-Adversarial Learning: Theory and Algorithms [82.97698406515667]
  Unsupervised domain adaptation is used in many machine learning applications where, during training, a model has access to unlabeled data in the target domain. We derive a novel generalization bound for domain adaptation that exploits a new measure of discrepancy between distributions based on a variational characterization of f-divergences.
  arXiv  Detail & Related papers  (2021-06-21T18:21:09Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Continual Learning for Natural Language Generation in Task-oriented Dialog Systems [72.92029584113676]
  Natural language generation (NLG) is an essential component of task-oriented dialog systems. We study NLG in a "continual learning" setting to expand its knowledge to new domains or functionalities incrementally. The major challenge towards this goal is catastrophic forgetting, meaning that a continually trained model tends to forget the knowledge it has learned before.
  arXiv  Detail & Related papers  (2020-10-02T10:32:29Z)
• Retrospective Loss: Looking Back to Improve Training of Deep Neural Networks [15.329684157845872]
  We introduce a new retrospective loss to improve the training of deep neural network models. Minimizing the retrospective loss, along with the task-specific loss, pushes the parameter state at the current training step towards the optimal parameter state. Although a simple idea, we analyze the method as well as to conduct comprehensive sets of experiments across domains.
  arXiv  Detail & Related papers  (2020-06-24T10:16:36Z)
• Adversarial-based neural networks for affect estimations in the wild [3.3335236123901995]
  In this work, we explore the use of latent features through our proposed adversarial-based networks for recognition in the wild. Specifically, our models operate by aggregating several modalities to our discriminator, which is further conditioned to the extracted latent features by the generator. Our experiments on the recently released SEWA dataset suggest the progressive improvements of our results.
  arXiv  Detail & Related papers  (2020-02-03T16:52:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.