Online Deep Learning from Doubly-Streaming Data

• URL: http://arxiv.org/abs/2204.11793v2
• Date: Wed, 27 Apr 2022 01:37:33 GMT
• Title: Online Deep Learning from Doubly-Streaming Data
• Authors: Heng Lian and John Scovil Atwood and Bojian Hou and Jian Wu and Yi He
• Abstract summary: This paper investigates a new online learning problem with doubly-streaming data, where the data streams are described by feature spaces that constantly evolve. A plausible idea to overcome the challenges is to establish relationship between the pre-and-post evolving feature spaces. We propose a novel OLD3S paradigm, where a shared latent subspace is discovered to summarize information from the old and new feature spaces.
• Score: 17.119725174036653
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper investigates a new online learning problem with doubly-streaming data, where the data streams are described by feature spaces that constantly evolve, with new features emerging and old features fading away. The challenges of this problem are two folds: 1) Data samples ceaselessly flowing in may carry shifted patterns over time, requiring learners to update hence adapt on-the-fly. 2) Newly emerging features are described by very few samples, resulting in weak learners that tend to make error predictions. A plausible idea to overcome the challenges is to establish relationship between the pre-and-post evolving feature spaces, so that an online learner can leverage the knowledge learned from the old features to better the learning performance on the new features. Unfortunately, this idea does not scale up to high-dimensional media streams with complex feature interplay, which suffers an tradeoff between onlineness (biasing shallow learners) and expressiveness(requiring deep learners). Motivated by this, we propose a novel OLD^3S paradigm, where a shared latent subspace is discovered to summarize information from the old and new feature spaces, building intermediate feature mapping relationship. A key trait of OLD^3S is to treat the model capacity as a learnable semantics, yields optimal model depth and parameters jointly, in accordance with the complexity and non-linearity of the input data streams in an online fashion. Both theoretical analyses and empirical studies substantiate the viability and effectiveness of our proposal.

Related papers

• Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning [79.46570165281084]
  We propose a Multi-Stage Knowledge Integration network (MulKI) to emulate the human learning process in distillation methods. MulKI achieves this through four stages, including Eliciting Ideas, Adding New Ideas, Distinguishing Ideas, and Making Connections. Our method demonstrates significant improvements in maintaining zero-shot capabilities while supporting continual learning across diverse downstream tasks.
  arXiv  Detail & Related papers  (2024-11-11T07:36:19Z)
• Function-space Parameterization of Neural Networks for Sequential Learning [22.095632118886225]
  Sequential learning paradigms pose challenges for gradient-based deep learning due to difficulties incorporating new data and retaining prior knowledge. We introduce a technique that converts neural networks from weight space to function space, through a dual parameterization. Our experiments demonstrate that we can retain knowledge in continual learning and incorporate new data efficiently.
  arXiv  Detail & Related papers  (2024-03-16T14:00:04Z)
• On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
  Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
  arXiv  Detail & Related papers  (2022-03-08T09:57:48Z)
• New Insights on Reducing Abrupt Representation Change in Online Continual Learning [69.05515249097208]
  We focus on the change in representations of observed data that arises when previously unobserved classes appear in the incoming data stream. We show that applying Experience Replay causes the newly added classes' representations to overlap significantly with the previous classes. We propose a new method which mitigates this issue by shielding the learned representations from drastic adaptation to accommodate new classes.
  arXiv  Detail & Related papers  (2022-03-08T01:37:00Z)
• Design of Explainability Module with Experts in the Loop for Visualization and Dynamic Adjustment of Continual Learning [5.039779583329608]
  Continual learning can enable neural networks to evolve by learning new tasks sequentially in task-changing scenarios. New novelties from the data stream in applications could contain anomalies that are meaningless for continual learning. We propose the conceptual design of an explainability module with experts in the loop based on techniques, such as dimension reduction, visualization, and evaluation strategies.
  arXiv  Detail & Related papers  (2022-02-14T15:00:22Z)
• Adaptive Online Incremental Learning for Evolving Data Streams [4.3386084277869505]
  The first major difficulty is concept drift, that is, the probability distribution in the streaming data would change as the data arrives. The second major difficulty is catastrophic forgetting, that is, forgetting what we have learned before when learning new knowledge. Our research builds on this observation and attempts to overcome these difficulties.
  arXiv  Detail & Related papers  (2022-01-05T14:25:53Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)
• Exploring Bayesian Deep Learning for Urgent Instructor Intervention Need in MOOC Forums [58.221459787471254]
  Massive Open Online Courses (MOOCs) have become a popular choice for e-learning thanks to their great flexibility. Due to large numbers of learners and their diverse backgrounds, it is taxing to offer real-time support. With the large volume of posts and high workloads for MOOC instructors, it is unlikely that the instructors can identify all learners requiring intervention. This paper explores for the first time Bayesian deep learning on learner-based text posts with two methods: Monte Carlo Dropout and Variational Inference.
  arXiv  Detail & Related papers  (2021-04-26T15:12:13Z)
• A case for new neural network smoothness constraints [34.373610792075205]
  We show that model smoothness is a useful inductive bias which aids generalization, adversarial robustness, generative modeling and reinforcement learning. We conclude that new advances in the field are hinging on finding ways to incorporate data, tasks and learning into our definitions of smoothness.
  arXiv  Detail & Related papers  (2020-12-14T22:07:32Z)
• Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams [42.525141660788]
  We introduce a system to enable learning and prediction at any point in time. In contrast to the major body of work in continual learning, data streams are processed in an online fashion. We obtain state-of-the-art performance by a significant margin on eight benchmarks, including three highly imbalanced data streams.
  arXiv  Detail & Related papers  (2020-09-02T09:39:26Z)
• Provable Meta-Learning of Linear Representations [114.656572506859]
  We provide fast, sample-efficient algorithms to address the dual challenges of learning a common set of features from multiple, related tasks, and transferring this knowledge to new, unseen tasks. We also provide information-theoretic lower bounds on the sample complexity of learning these linear features.
  arXiv  Detail & Related papers  (2020-02-26T18:21:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.