Federated Non-negative Matrix Factorization for Short Texts Topic Modeling with Mutual Information

• URL: http://arxiv.org/abs/2205.13300v1
• Date: Thu, 26 May 2022 12:22:34 GMT
• Title: Federated Non-negative Matrix Factorization for Short Texts Topic Modeling with Mutual Information
• Authors: Shijing Si, Jianzong Wang, Ruiyi Zhang, Qinliang Su and Jing Xiao
• Abstract summary: This paper proposes a Federated NMF (FedNMF) framework, which allows multiple clients to collaboratively train a high-quality NMF based topic model with locally stored data. Experimental results show that our FedNMF+MI methods outperform Federated Latent Dirichlet Allocation (FedLDA) and the FedNMF without MI methods for short texts.
• Score: 43.012719398648144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Non-negative matrix factorization (NMF) based topic modeling is widely used in natural language processing (NLP) to uncover hidden topics of short text documents. Usually, training a high-quality topic model requires large amount of textual data. In many real-world scenarios, customer textual data should be private and sensitive, precluding uploading to data centers. This paper proposes a Federated NMF (FedNMF) framework, which allows multiple clients to collaboratively train a high-quality NMF based topic model with locally stored data. However, standard federated learning will significantly undermine the performance of topic models in downstream tasks (e.g., text classification) when the data distribution over clients is heterogeneous. To alleviate this issue, we further propose FedNMF+MI, which simultaneously maximizes the mutual information (MI) between the count features of local texts and their topic weight vectors to mitigate the performance degradation. Experimental results show that our FedNMF+MI methods outperform Federated Latent Dirichlet Allocation (FedLDA) and the FedNMF without MI methods for short texts by a significant margin on both coherence score and classification F1 score.

Related papers

• GloCOM: A Short Text Neural Topic Model via Global Clustering Context [29.685615665355396]
  GloCOM is a novel model for constructing global clustering contexts for short documents. It infers both global topic distributions for clustering contexts and local distributions for individual short texts. Our approach outperforms other state-of-the-art models in both topic quality and document representations.
  arXiv  Detail & Related papers  (2024-11-30T16:19:25Z)
• TopicTag: Automatic Annotation of NMF Topic Models Using Chain of Thought and Prompt Tuning with LLMs [1.1826529992155377]
  Non-negative matrix factorization (NMF) is a common unsupervised approach that decomposes a term frequency-inverse document frequency (TF-IDF) matrix to uncover latent topics. We present a methodology for automating topic labeling in documents clustered via NMF with automatic model determination (NMFk) By leveraging the output of NMFk and employing prompt engineering, we utilize large language models (LLMs) to generate accurate topic labels.
  arXiv  Detail & Related papers  (2024-07-29T00:18:17Z)
• Federated Binary Matrix Factorization using Proximal Optimization [43.01276597844757]
  In this paper, we adapt a state-of-the-art binary matrix factorization relaxation to BMF. We show that our algorithm outperforms, in terms of quality and efficacy, federation schemes of state-of-the-art BMF methods on a diverse set of real-world and synthetic data.
  arXiv  Detail & Related papers  (2024-07-01T20:10:24Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction [49.510163437116645]
  Click-through rate (CTR) prediction plays as a core function module in personalized online services. Traditional ID-based models for CTR prediction take as inputs the one-hot encoded ID features of tabular modality. Pretrained Language Models(PLMs) has given rise to another paradigm, which takes as inputs the sentences of textual modality. We propose to conduct Fine-grained feature-level ALignment between ID-based Models and Pretrained Language Models(FLIP) for CTR prediction.
  arXiv  Detail & Related papers  (2023-10-30T11:25:03Z)
• Federated Learning with Neural Graphical Models [2.2721854258621064]
  Federated Learning (FL) addresses the need to create models based on proprietary data. We develop a FL framework which maintains a global NGM model that learns the averaged information from the local NGM models. We experimentally demonstrated the use of FedNGMs for extracting insights from CDC's Infant Mortality dataset.
  arXiv  Detail & Related papers  (2023-09-20T23:24:22Z)
• Personalized Federated Learning via Gradient Modulation for Heterogeneous Text Summarization [21.825321314169642]
  We propose a federated learning text summarization scheme, which allows users to share the global model in a cooperative learning manner without sharing raw data. FedSUMM can achieve faster model convergence on PFL algorithm for task-specific text summarization.
  arXiv  Detail & Related papers  (2023-04-23T03:18:46Z)
• FedMix: Approximation of Mixup under Mean Augmented Federated Learning [60.503258658382]
  Federated learning (FL) allows edge devices to collectively learn a model without directly sharing data within each device. Current state-of-the-art algorithms suffer from performance degradation as the heterogeneity of local data across clients increases. We propose a new augmentation algorithm, named FedMix, which is inspired by a phenomenal yet simple data augmentation method, Mixup.
  arXiv  Detail & Related papers  (2021-07-01T06:14:51Z)
• FedNLP: A Research Platform for Federated Learning in Natural Language Processing [55.01246123092445]
  We present the FedNLP, a research platform for federated learning in NLP. FedNLP supports various popular task formulations in NLP such as text classification, sequence tagging, question answering, seq2seq generation, and language modeling. Preliminary experiments with FedNLP reveal that there exists a large performance gap between learning on decentralized and centralized datasets.
  arXiv  Detail & Related papers  (2021-04-18T11:04:49Z)
• Fast and Secure Distributed Nonnegative Matrix Factorization [13.672004396034856]
  Nonnegative matrix factorization (NMF) has been successfully applied in several data mining tasks. We study the acceleration and security problems of distributed NMF.
  arXiv  Detail & Related papers  (2020-09-07T01:12:20Z)
• Towards Making the Most of Context in Neural Machine Translation [112.9845226123306]
  We argue that previous research did not make a clear use of the global context. We propose a new document-level NMT framework that deliberately models the local context of each sentence.
  arXiv  Detail & Related papers  (2020-02-19T03:30:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.