Context-tree weighting for real-valued time series: Bayesian inference with hierarchical mixture models

• URL: http://arxiv.org/abs/2106.03023v4
• Date: Sat, 15 Apr 2023 02:14:38 GMT
• Title: Context-tree weighting for real-valued time series: Bayesian inference with hierarchical mixture models
• Authors: Ioannis Papageorgiou, Ioannis Kontoyiannis
• Abstract summary: A general, hierarchical Bayesian modelling framework is developed for building mixture models for times series. This development is based, in part, on the use of context trees, and it includes a collection of effective algorithmic tools for learning and inference. The utility of the general framework is illustrated in detail when autoregressive (AR) models are used at the bottom level, resulting in a nonlinear AR mixture model.
• Score: 8.37609145576126
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-valued time series are ubiquitous in the sciences and engineering. In this work, a general, hierarchical Bayesian modelling framework is developed for building mixture models for times series. This development is based, in part, on the use of context trees, and it includes a collection of effective algorithmic tools for learning and inference. A discrete context (or 'state') is extracted for each sample, consisting of a discretised version of some of the most recent observations preceding it. The set of all relevant contexts are represented as a discrete context-tree. At the bottom level, a different real-valued time series model is associated with each context-state, i.e., with each leaf of the tree. This defines a very general framework that can be used in conjunction with any existing model class to build flexible and interpretable mixture models. Extending the idea of context-tree weighting leads to algorithms that allow for efficient, exact Bayesian inference in this setting. The utility of the general framework is illustrated in detail when autoregressive (AR) models are used at the bottom level, resulting in a nonlinear AR mixture model. The associated methods are found to outperform several state-of-the-art techniques on simulated and real-world experiments.

Related papers

• How compositional generalization and creativity improve as diffusion models are trained [82.08869888944324]
  How many samples do generative models need to learn the composition rules, so as to produce a number of novel data? We consider diffusion models trained on simple context-free grammars - tree-like graphical models used to represent the structure of data such as language and images. We demonstrate that diffusion models learn compositional rules with the sample complexity required for clustering features with statistically similar context, a process similar to the word2vec.
  arXiv  Detail & Related papers  (2025-02-17T18:06:33Z)
• Approximate learning of parsimonious Bayesian context trees [0.0]
  The proposed framework is tested on synthetic and real-world data examples. It outperforms existing sequence models when fitted to real protein sequences and honeypot computer terminal sessions.
  arXiv  Detail & Related papers  (2024-07-27T11:50:40Z)
• A Unified Approach to Extract Interpretable Rules from Tree Ensembles via Integer Programming [2.1408617023874443]
  Tree ensembles are very popular machine learning models, known for their effectiveness in supervised classification and regression tasks. Our work aims to extract an optimized list of rules from a trained tree ensemble, providing the user with a condensed, interpretable model that retains most of the predictive power of the full model. Our extensive computational experiments offer statistically significant evidence that our method is competitive with other rule extraction methods in terms of predictive performance and fidelity towards the tree ensemble.
  arXiv  Detail & Related papers  (2024-06-30T22:33:47Z)
• The Bayesian Context Trees State Space Model for time series modelling and forecasting [8.37609145576126]
  A hierarchical Bayesian framework is introduced for developing rich mixture models for real-valued time series. At the top level, meaningful discrete states are identified as appropriately quantised values of some of the most recent samples. At the bottom level, a different, arbitrary model for real-valued time series - a base model - is associated with each state.
  arXiv  Detail & Related papers  (2023-08-02T02:40:42Z)
• Syntactically Robust Training on Partially-Observed Data for Open Information Extraction [25.59133746149343]
  Open Information Extraction models have shown promising results with sufficient supervision. We propose a syntactically robust training framework that enables models to be trained on a syntactic-abundant distribution.
  arXiv  Detail & Related papers  (2023-01-17T12:39:13Z)
• SETAR-Tree: A Novel and Accurate Tree Algorithm for Global Time Series Forecasting [7.206754802573034]
  In this paper, we explore the close connections between TAR models and regression trees. We introduce a new forecasting-specific tree algorithm that trains global Pooled Regression (PR) models in the leaves. In our evaluation, the proposed tree and forest models are able to achieve significantly higher accuracy than a set of state-of-the-art tree-based algorithms.
  arXiv  Detail & Related papers  (2022-11-16T04:30:42Z)
• On the Compositional Generalization Gap of In-Context Learning [73.09193595292233]
  We look at the gap between the in-distribution (ID) and out-of-distribution (OOD) performance of such models in semantic parsing tasks with in-context learning. We evaluate four model families, OPT, BLOOM, CodeGen and Codex on three semantic parsing datasets.
  arXiv  Detail & Related papers  (2022-11-15T19:56:37Z)
• Partial Order in Chaos: Consensus on Feature Attributions in the Rashomon Set [50.67431815647126]
  Post-hoc global/local feature attribution methods are being progressively employed to understand machine learning models. We show that partial orders of local/global feature importance arise from this methodology. We show that every relation among features present in these partial orders also holds in the rankings provided by existing approaches.
  arXiv  Detail & Related papers  (2021-10-26T02:53:14Z)
• Improving Label Quality by Jointly Modeling Items and Annotators [68.8204255655161]
  We propose a fully Bayesian framework for learning ground truth labels from noisy annotators. Our framework ensures scalability by factoring a generative, Bayesian soft clustering model over label distributions into the classic David and Skene joint annotator-data model.
  arXiv  Detail & Related papers  (2021-06-20T02:15:20Z)
• Evaluating the Disentanglement of Deep Generative Models through Manifold Topology [66.06153115971732]
  We present a method for quantifying disentanglement that only uses the generative model. We empirically evaluate several state-of-the-art models across multiple datasets.
  arXiv  Detail & Related papers  (2020-06-05T20:54:11Z)
• On the Discrepancy between Density Estimation and Sequence Generation [92.70116082182076]
  log-likelihood is highly correlated with BLEU when we consider models within the same family. We observe no correlation between rankings of models across different families.
  arXiv  Detail & Related papers  (2020-02-17T20:13:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.