ALBU: An approximate Loopy Belief message passing algorithm for LDA to
improve performance on small data sets
- URL: http://arxiv.org/abs/2110.00635v1
- Date: Fri, 1 Oct 2021 19:55:12 GMT
- Title: ALBU: An approximate Loopy Belief message passing algorithm for LDA to
improve performance on small data sets
- Authors: Rebecca M.C. Taylor and Johan A. du Preez
- Abstract summary: We present a novel variational message passing algorithm as applied to Latent Dirichlet Allocation (LDA)
We compare it with the gold standard VB and collapsed Gibbs sampling algorithms.
Using coherence measures for the text corpora and KLD with the simulations we show that ALBU learns latent distributions more accurately than does VB.
- Score: 3.5027291542274366
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Variational Bayes (VB) applied to latent Dirichlet allocation (LDA) has
become the most popular algorithm for aspect modeling. While sufficiently
successful in text topic extraction from large corpora, VB is less successful
in identifying aspects in the presence of limited data. We present a novel
variational message passing algorithm as applied to Latent Dirichlet Allocation
(LDA) and compare it with the gold standard VB and collapsed Gibbs sampling. In
situations where marginalisation leads to non-conjugate messages, we use ideas
from sampling to derive approximate update equations. In cases where conjugacy
holds, Loopy Belief update (LBU) (also known as Lauritzen-Spiegelhalter) is
used. Our algorithm, ALBU (approximate LBU), has strong similarities with
Variational Message Passing (VMP) (which is the message passing variant of VB).
To compare the performance of the algorithms in the presence of limited data,
we use data sets consisting of tweets and news groups. Additionally, to perform
more fine grained evaluations and comparisons, we use simulations that enable
comparisons with the ground truth via Kullback-Leibler divergence (KLD). Using
coherence measures for the text corpora and KLD with the simulations we show
that ALBU learns latent distributions more accurately than does VB, especially
for smaller data sets.
Related papers
- ExaRanker-Open: Synthetic Explanation for IR using Open-Source LLMs [60.81649785463651]
We introduce ExaRanker-Open, where we adapt and explore the use of open-source language models to generate explanations.
Our findings reveal that incorporating explanations consistently enhances neural rankers, with benefits escalating as the LLM size increases.
arXiv Detail & Related papers (2024-02-09T11:23:14Z) - MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering [64.6741991162092]
We present MinPrompt, a minimal data augmentation framework for open-domain question answering.
We transform the raw text into a graph structure to build connections between different factual sentences.
We then apply graph algorithms to identify the minimal set of sentences needed to cover the most information in the raw text.
We generate QA pairs based on the identified sentence subset and train the model on the selected sentences to obtain the final model.
arXiv Detail & Related papers (2023-10-08T04:44:36Z) - Performance Evaluation and Comparison of a New Regression Algorithm [4.125187280299247]
We compare the performance of a newly proposed regression algorithm against four conventional machine learning algorithms.
The reader is free to replicate our results since we have provided the source code in a GitHub repository.
arXiv Detail & Related papers (2023-06-15T13:01:16Z) - SimLDA: A tool for topic model evaluation [2.6397379133308214]
We present a novel variational message passing algorithm as applied to Latent Dirichlet Allocation (LDA)
We compare it with the gold standard VB and collapsed Gibbs sampling algorithms.
Using coherence measures we show that ALBU learns latent distributions more accurately than does VB, especially for smaller data sets.
arXiv Detail & Related papers (2022-08-19T12:25:53Z) - A Bayesian Bradley-Terry model to compare multiple ML algorithms on
multiple data sets [4.394728504061753]
This paper proposes a Bayesian model to compare multiple algorithms on multiple data sets, on any metric.
The model is based on the Bradley-Terry model, that counts the number of times one algorithm performs better than another on different data sets.
arXiv Detail & Related papers (2022-08-09T17:59:06Z) - Langevin Monte Carlo for Contextual Bandits [72.00524614312002]
Langevin Monte Carlo Thompson Sampling (LMC-TS) is proposed to directly sample from the posterior distribution in contextual bandits.
We prove that the proposed algorithm achieves the same sublinear regret bound as the best Thompson sampling algorithms for a special case of contextual bandits.
arXiv Detail & Related papers (2022-06-22T17:58:23Z) - Can Active Learning Preemptively Mitigate Fairness Issues? [66.84854430781097]
dataset bias is one of the prevailing causes of unfairness in machine learning.
We study whether models trained with uncertainty-based ALs are fairer in their decisions with respect to a protected class.
We also explore the interaction of algorithmic fairness methods such as gradient reversal (GRAD) and BALD.
arXiv Detail & Related papers (2021-04-14T14:20:22Z) - Sparse Feature Selection Makes Batch Reinforcement Learning More Sample
Efficient [62.24615324523435]
This paper provides a statistical analysis of high-dimensional batch Reinforcement Learning (RL) using sparse linear function approximation.
When there is a large number of candidate features, our result sheds light on the fact that sparsity-aware methods can make batch RL more sample efficient.
arXiv Detail & Related papers (2020-11-08T16:48:02Z) - DecAug: Augmenting HOI Detection via Decomposition [54.65572599920679]
Current algorithms suffer from insufficient training samples and category imbalance within datasets.
We propose an efficient and effective data augmentation method called DecAug for HOI detection.
Experiments show that our method brings up to 3.3 mAP and 1.6 mAP improvements on V-COCO and HICODET dataset.
arXiv Detail & Related papers (2020-10-02T13:59:05Z) - Active Sampling for Pairwise Comparisons via Approximate Message Passing
and Information Gain Maximization [5.771869590520189]
We propose ASAP, an active sampling algorithm based on approximate message passing and expected information gain.
We show that ASAP offers the highest accuracy of inferred scores compared to the existing methods.
arXiv Detail & Related papers (2020-04-12T20:48:10Z) - Improving Reliability of Latent Dirichlet Allocation by Assessing Its
Stability Using Clustering Techniques on Replicated Runs [0.3499870393443268]
We study the stability of LDA by comparing assignments from replicated runs.
We propose to quantify the similarity of two generated topics by a modified Jaccard coefficient.
We show that the measure S-CLOP is useful for assessing the stability of LDA models.
arXiv Detail & Related papers (2020-02-14T07:10:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.