Online Deep Learning based on Auto-Encoder
- URL: http://arxiv.org/abs/2201.07383v1
- Date: Wed, 19 Jan 2022 02:14:57 GMT
- Title: Online Deep Learning based on Auto-Encoder
- Authors: Si-si Zhang, Jian-wei Liu, Xin Zuo, Run-kun Lu, Si-ming Lian
- Abstract summary: We propose a two-phase Online Deep Learning based on Auto-Encoder (ODLAE)
Based on auto-encoder, considering reconstruction loss, we extract abstract hierarchical latent representations of instances.
We devise two fusion strategies: the output-level fusion strategy, which is obtained by fusing the classification results of each hidden layer; and feature-level fusion strategy, which is leveraged self-attention mechanism to fusion every hidden layer output.
- Score: 4.128388784932455
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Online learning is an important technical means for sketching massive
real-time and high-speed data. Although this direction has attracted intensive
attention, most of the literature in this area ignore the following three
issues: (1) they think little of the underlying abstract hierarchical latent
information existing in examples, even if extracting these abstract
hierarchical latent representations is useful to better predict the class
labels of examples; (2) the idea of preassigned model on unseen datapoints is
not suitable for modeling streaming data with evolving probability
distribution. This challenge is referred as model flexibility. And so, with
this in minds, the online deep learning model we need to design should have a
variable underlying structure; (3) moreover, it is of utmost importance to
fusion these abstract hierarchical latent representations to achieve better
classification performance, and we should give different weights to different
levels of implicit representation information when dealing with the data
streaming where the data distribution changes. To address these issues, we
propose a two-phase Online Deep Learning based on Auto-Encoder (ODLAE). Based
on auto-encoder, considering reconstruction loss, we extract abstract
hierarchical latent representations of instances; Based on predictive loss, we
devise two fusion strategies: the output-level fusion strategy, which is
obtained by fusing the classification results of encoder each hidden layer; and
feature-level fusion strategy, which is leveraged self-attention mechanism to
fusion every hidden layer output. Finally, in order to improve the robustness
of the algorithm, we also try to utilize the denoising auto-encoder to yield
hierarchical latent representations. Experimental results on different datasets
are presented to verify the validity of our proposed algorithm (ODLAE)
outperforms several baselines.
Related papers
- Language Models are Graph Learners [70.14063765424012]
Language Models (LMs) are challenging the dominance of domain-specific models, including Graph Neural Networks (GNNs) and Graph Transformers (GTs)
We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art GNNs on node classification tasks.
arXiv Detail & Related papers (2024-10-03T08:27:54Z) - OPUS: Occupancy Prediction Using a Sparse Set [64.60854562502523]
We present a framework to simultaneously predict occupied locations and classes using a set of learnable queries.
OPUS incorporates a suite of non-trivial strategies to enhance model performance.
Our lightest model achieves superior RayIoU on the Occ3D-nuScenes dataset at near 2x FPS, while our heaviest model surpasses previous best results by 6.1 RayIoU.
arXiv Detail & Related papers (2024-09-14T07:44:22Z) - Statistical signatures of abstraction in deep neural networks [0.0]
We study how abstract representations emerge in a Deep Belief Network (DBN) trained on benchmark datasets.
We show that the representation approaches an universal model determined by the principle of maximal relevance.
We also show that plasticity increases with depth, in a similar way as it does in the brain.
arXiv Detail & Related papers (2024-07-01T14:13:11Z) - How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Model [4.215221129670858]
We show that by introducing sparsity to generative hierarchical models of data, the task acquires insensitivity to spatial transformations that are discrete versions of smooth transformations.
We quantify how the sample complexity of CNNs learning the SRHM depends on both the sparsity and hierarchical structure of the task.
arXiv Detail & Related papers (2024-04-16T17:01:27Z) - The Trifecta: Three simple techniques for training deeper
Forward-Forward networks [0.0]
We propose a collection of three techniques that synergize exceptionally well and drastically improve the Forward-Forward algorithm on deeper networks.
Our experiments demonstrate that our models are on par with similarly structured, backpropagation-based models in both training speed and test accuracy on simple datasets.
arXiv Detail & Related papers (2023-11-29T22:44:32Z) - Homological Convolutional Neural Networks [4.615338063719135]
We propose a novel deep learning architecture that exploits the data structural organization through topologically constrained network representations.
We test our model on 18 benchmark datasets against 5 classic machine learning and 3 deep learning models.
arXiv Detail & Related papers (2023-08-26T08:48:51Z) - Towards Automated Imbalanced Learning with Deep Hierarchical
Reinforcement Learning [57.163525407022966]
Imbalanced learning is a fundamental challenge in data mining, where there is a disproportionate ratio of training samples in each class.
Over-sampling is an effective technique to tackle imbalanced learning through generating synthetic samples for the minority class.
We propose AutoSMOTE, an automated over-sampling algorithm that can jointly optimize different levels of decisions.
arXiv Detail & Related papers (2022-08-26T04:28:01Z) - Imposing Consistency for Optical Flow Estimation [73.53204596544472]
Imposing consistency through proxy tasks has been shown to enhance data-driven learning.
This paper introduces novel and effective consistency strategies for optical flow estimation.
arXiv Detail & Related papers (2022-04-14T22:58:30Z) - Weakly Supervised Change Detection Using Guided Anisotropic Difusion [97.43170678509478]
We propose original ideas that help us to leverage such datasets in the context of change detection.
First, we propose the guided anisotropic diffusion (GAD) algorithm, which improves semantic segmentation results.
We then show its potential in two weakly-supervised learning strategies tailored for change detection.
arXiv Detail & Related papers (2021-12-31T10:03:47Z) - Deep Fusion Clustering Network [38.540761683389135]
We propose a Deep Fusion Clustering Network (DFCN) for deep clustering.
In our network, an interdependency learning-based Structure and Attribute Information Fusion (SAIF) module is proposed to explicitly merge the representations learned by an autoencoder and a graph autoencoder.
Experiments on six benchmark datasets have demonstrated that the proposed DFCN consistently outperforms the state-of-the-art deep clustering methods.
arXiv Detail & Related papers (2020-12-15T09:37:59Z) - Relation-Guided Representation Learning [53.60351496449232]
We propose a new representation learning method that explicitly models and leverages sample relations.
Our framework well preserves the relations between samples.
By seeking to embed samples into subspace, we show that our method can address the large-scale and out-of-sample problem.
arXiv Detail & Related papers (2020-07-11T10:57:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.