Lifelong Generative Modelling Using Dynamic Expansion Graph Model
- URL: http://arxiv.org/abs/2112.08370v1
- Date: Wed, 15 Dec 2021 17:35:27 GMT
- Title: Lifelong Generative Modelling Using Dynamic Expansion Graph Model
- Authors: Fei Ye and Adrian G. Bors
- Abstract summary: We study the forgetting behaviour of VAEs using a joint GR and ENA methodology.
We propose a novel Dynamic Expansion Graph Model (DEGM)
- Score: 15.350366047108103
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Variational Autoencoders (VAEs) suffer from degenerated performance, when
learning several successive tasks. This is caused by catastrophic forgetting.
In order to address the knowledge loss, VAEs are using either Generative Replay
(GR) mechanisms or Expanding Network Architectures (ENA). In this paper we
study the forgetting behaviour of VAEs using a joint GR and ENA methodology, by
deriving an upper bound on the negative marginal log-likelihood. This
theoretical analysis provides new insights into how VAEs forget the previously
learnt knowledge during lifelong learning. The analysis indicates the best
performance achieved when considering model mixtures, under the ENA framework,
where there are no restrictions on the number of components. However, an
ENA-based approach may require an excessive number of parameters. This
motivates us to propose a novel Dynamic Expansion Graph Model (DEGM). DEGM
expands its architecture, according to the novelty associated with each new
databases, when compared to the information already learnt by the network from
previous tasks. DEGM training optimizes knowledge structuring, characterizing
the joint probabilistic representations corresponding to the past and more
recently learned tasks. We demonstrate that DEGM guarantees optimal performance
for each task while also minimizing the required number of parameters.
Supplementary materials (SM) and source code are available in
https://github.com/dtuzi123/Expansion-Graph-Model.
Related papers
- GraphFM: A Comprehensive Benchmark for Graph Foundation Model [33.157367455390144]
Foundation Models (FMs) serve as a general class for the development of artificial intelligence systems.
Despite extensive research into self-supervised learning as the cornerstone of FMs, several outstanding issues persist.
The extent of generalization capability on downstream tasks remains unclear.
It is unknown how effectively these models can scale to large datasets.
arXiv Detail & Related papers (2024-06-12T15:10:44Z) - ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained
Language Models for Question Answering over Knowledge Graph [142.42275983201978]
We propose a subgraph-aware self-attention mechanism to imitate the GNN for performing structured reasoning.
We also adopt an adaptation tuning strategy to adapt the model parameters with 20,000 subgraphs with synthesized questions.
Experiments show that ReasoningLM surpasses state-of-the-art models by a large margin, even with fewer updated parameters and less training data.
arXiv Detail & Related papers (2023-12-30T07:18:54Z) - DORE: Document Ordered Relation Extraction based on Generative Framework [56.537386636819626]
This paper investigates the root cause of the underwhelming performance of the existing generative DocRE models.
We propose to generate a symbolic and ordered sequence from the relation matrix which is deterministic and easier for model to learn.
Experimental results on four datasets show that our proposed method can improve the performance of the generative DocRE models.
arXiv Detail & Related papers (2022-10-28T11:18:10Z) - uGLAD: Sparse graph recovery by optimizing deep unrolled networks [11.48281545083889]
We present a novel technique to perform sparse graph recovery by optimizing deep unrolled networks.
Our model, uGLAD, builds upon and extends the state-of-the-art model GLAD to the unsupervised setting.
We evaluate model results on synthetic Gaussian data, non-Gaussian data generated from Gene Regulatory Networks, and present a case study in anaerobic digestion.
arXiv Detail & Related papers (2022-05-23T20:20:27Z) - An Expectation-Maximization Perspective on Federated Learning [75.67515842938299]
Federated learning describes the distributed training of models across multiple clients while keeping the data private on-device.
In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters.
We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting.
arXiv Detail & Related papers (2021-11-19T12:58:59Z) - Causal Incremental Graph Convolution for Recommender System Retraining [89.25922726558875]
Real-world recommender system needs to be regularly retrained to keep with the new data.
In this work, we consider how to efficiently retrain graph convolution network (GCN) based recommender models.
arXiv Detail & Related papers (2021-08-16T04:20:09Z) - Shared and Private VAEs with Generative Replay for Continual Learning [1.90365714903665]
Continual learning tries to learn new tasks without forgetting previously learned ones.
Most of the existing artificial neural network(ANN) models fail, while humans do the same by remembering previous works throughout their life.
We show our hybrid model effectively avoids forgetting and achieves state-of-the-art results on visual continual learning benchmarks such as MNIST, Permuted MNIST(QMNIST), CIFAR100, and miniImageNet datasets.
arXiv Detail & Related papers (2021-05-17T06:18:36Z) - Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
Variational autoencoders (VAE) are a powerful and widely-used class of generative models.
We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs.
Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
arXiv Detail & Related papers (2021-01-06T17:36:26Z) - Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic
Parsing [85.35582118010608]
Task-oriented semantic parsing is a critical component of virtual assistants.
Recent advances in deep learning have enabled several approaches to successfully parse more complex queries.
We propose a novel method that outperforms a supervised neural model at a 10-fold data reduction.
arXiv Detail & Related papers (2020-10-07T17:47:53Z) - Multi-view Self-Constructing Graph Convolutional Networks with Adaptive
Class Weighting Loss for Semantic Segmentation [23.623276007011373]
We propose a novel architecture called the Multi-view Self-Constructing Graph Convolutional Networks (MSCG-Net) for semantic segmentation.
We leverage multiple views in order to explicitly exploit the rotational invariance in airborne images.
We demonstrate the effectiveness and flexibility of the proposed method on the Agriculture-Vision challenge and our model achieves very competitive results.
arXiv Detail & Related papers (2020-04-21T22:18:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.