T-SaS: Toward Shift-aware Dynamic Adaptation for Streaming Data
- URL: http://arxiv.org/abs/2309.02610v1
- Date: Tue, 5 Sep 2023 22:55:10 GMT
- Title: T-SaS: Toward Shift-aware Dynamic Adaptation for Streaming Data
- Authors: Weijieying Ren, Tianxiang Zhao, Wei Qin, Kunpeng Liu
- Abstract summary: This paper aims to solve the problem of sequential data modeling in the presence of sudden distribution shifts.
Specifically, we design a Bayesian framework, dubbed as T-SaS, with a discrete distribution-modeling variable to capture abrupt shifts of data.
The proposed method learns specific model parameters for each distribution by learning which neurons should be activated in the full network.
- Score: 9.829993835712422
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: In many real-world scenarios, distribution shifts exist in the streaming data
across time steps. Many complex sequential data can be effectively divided into
distinct regimes that exhibit persistent dynamics. Discovering the shifted
behaviors and the evolving patterns underlying the streaming data are important
to understand the dynamic system. Existing methods typically train one robust
model to work for the evolving data of distinct distributions or sequentially
adapt the model utilizing explicitly given regime boundaries. However, there
are two challenges: (1) shifts in data streams could happen drastically and
abruptly without precursors. Boundaries of distribution shifts are usually
unavailable, and (2) training a shared model for all domains could fail to
capture varying patterns. This paper aims to solve the problem of sequential
data modeling in the presence of sudden distribution shifts that occur without
any precursors. Specifically, we design a Bayesian framework, dubbed as T-SaS,
with a discrete distribution-modeling variable to capture abrupt shifts of
data. Then, we design a model that enable adaptation with dynamic network
selection conditioned on that discrete variable. The proposed method learns
specific model parameters for each distribution by learning which neurons
should be activated in the full network. A dynamic masking strategy is adopted
here to support inter-distribution transfer through the overlapping of a set of
sparse networks. Extensive experiments show that our proposed method is
superior in both accurately detecting shift boundaries to get segments of
varying distributions and effectively adapting to downstream forecast or
classification tasks.
Related papers
- Constrained Diffusion Models via Dual Training [80.03953599062365]
We develop constrained diffusion models based on desired distributions informed by requirements.
We show that our constrained diffusion models generate new data from a mixture data distribution that achieves the optimal trade-off among objective and constraints.
arXiv Detail & Related papers (2024-08-27T14:25:42Z) - Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion [61.03681839276652]
Diffusion Forcing is a new training paradigm where a diffusion model is trained to denoise a set of tokens with independent per-token noise levels.
We apply Diffusion Forcing to sequence generative modeling by training a causal next-token prediction model to generate one or several future tokens.
arXiv Detail & Related papers (2024-07-01T15:43:25Z) - Learning Divergence Fields for Shift-Robust Graph Representations [73.11818515795761]
In this work, we propose a geometric diffusion model with learnable divergence fields for the challenging problem with interdependent data.
We derive a new learning objective through causal inference, which can guide the model to learn generalizable patterns of interdependence that are insensitive across domains.
arXiv Detail & Related papers (2024-06-07T14:29:21Z) - Training Implicit Generative Models via an Invariant Statistical Loss [3.139474253994318]
Implicit generative models have the capability to learn arbitrary complex data distributions.
On the downside, training requires telling apart real data from artificially-generated ones using adversarial discriminators.
We develop a discriminator-free method for training one-dimensional (1D) generative implicit models.
arXiv Detail & Related papers (2024-02-26T09:32:28Z) - Out-of-Distribution Generalized Dynamic Graph Neural Network with
Disentangled Intervention and Invariance Promotion [61.751257172868186]
Dynamic graph neural networks (DyGNNs) have demonstrated powerful predictive abilities by exploiting graph and temporal dynamics.
Existing DyGNNs fail to handle distribution shifts, which naturally exist in dynamic graphs.
arXiv Detail & Related papers (2023-11-24T02:42:42Z) - Adaptive Test-Time Personalization for Federated Learning [51.25437606915392]
We introduce a novel setting called test-time personalized federated learning (TTPFL)
In TTPFL, clients locally adapt a global model in an unsupervised way without relying on any labeled data during test-time.
We propose a novel algorithm called ATP to adaptively learn the adaptation rates for each module in the model from distribution shifts among source domains.
arXiv Detail & Related papers (2023-10-28T20:42:47Z) - Bayesian Flow Networks [4.585102332532472]
This paper introduces Bayesian Flow Networks (BFNs), a new class of generative model in which the parameters of a set of independent distributions are modified with Bayesian inference.
Starting from a simple prior and iteratively updating the two distributions yields a generative procedure similar to the reverse process of diffusion models.
BFNs achieve competitive log-likelihoods for image modelling on dynamically binarized MNIST and CIFAR-10, and outperform all known discrete diffusion models on the text8 character-level language modelling task.
arXiv Detail & Related papers (2023-08-14T09:56:35Z) - Can Push-forward Generative Models Fit Multimodal Distributions? [3.8615905456206256]
We show that the Lipschitz constant of generative networks has to be large in order to fit multimodal distributions.
We validate our findings on one-dimensional and image datasets and empirically show that generative models consisting of stacked networks with input at each step do not suffer of such limitations.
arXiv Detail & Related papers (2022-06-29T09:03:30Z) - Conditional Permutation Invariant Flows [23.740061786510417]
We present a conditional generative probabilistic model of set-valued data with a tractable log density.
These dynamics are driven by a learnable per-set-element term and pairwise interactions, both parametrized by deep neural networks.
We illustrate the utility of this model via applications including (1) complex traffic scene generation conditioned on visually specified map information, and (2) object bounding box generation conditioned directly on images.
arXiv Detail & Related papers (2022-06-17T21:43:38Z) - Variational Mixture of Normalizing Flows [0.0]
Deep generative models, such as generative adversarial networks autociteGAN, variational autoencoders autocitevaepaper, and their variants, have seen wide adoption for the task of modelling complex data distributions.
Normalizing flows have overcome this limitation by leveraging the change-of-suchs formula for probability density functions.
The present work overcomes this by using normalizing flows as components in a mixture model and devising an end-to-end training procedure for such a model.
arXiv Detail & Related papers (2020-09-01T17:20:08Z) - Pre-Trained Models for Heterogeneous Information Networks [57.78194356302626]
We propose a self-supervised pre-training and fine-tuning framework, PF-HIN, to capture the features of a heterogeneous information network.
PF-HIN consistently and significantly outperforms state-of-the-art alternatives on each of these tasks, on four datasets.
arXiv Detail & Related papers (2020-07-07T03:36:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.