Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis

• URL: http://arxiv.org/abs/2405.20602v2
• Date: Mon, 19 Aug 2024 01:36:45 GMT
• Title: Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis
• Authors: Seunghwan An, Gyeongdong Woo, Jaesung Lim, ChangHyun Kim, Sungchul Hong, Jong-June Jeon,
• Abstract summary: We introduce MaCoDE by redefining the consecutive multi-class classification task of Masked Modeling (MLM) as histogram-based conditional density estimation. Our approach enables the estimation of conditional densities across arbitrary combinations of target and conditional variables. To validate our proposed model, we evaluate its performance in synthetic data generation across 10 real-world datasets.
• Score: 0.74454067778951
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, our goal is to generate synthetic data for heterogeneous (mixed-type) tabular datasets with high machine learning utility (MLu). Since the MLu performance depends on accurately approximating the conditional distributions, we focus on devising a synthetic data generation method based on conditional distribution estimation. We introduce MaCoDE by redefining the consecutive multi-class classification task of Masked Language Modeling (MLM) as histogram-based non-parametric conditional density estimation. Our approach enables the estimation of conditional densities across arbitrary combinations of target and conditional variables. We bridge the theoretical gap between distributional learning and MLM by demonstrating that minimizing the orderless multi-class classification loss leads to minimizing the total variation distance between conditional distributions. To validate our proposed model, we evaluate its performance in synthetic data generation across 10 real-world datasets, demonstrating its ability to adjust data privacy levels easily without re-training. Additionally, since masked input tokens in MLM are analogous to missing data, we further assess its effectiveness in handling training datasets with missing values, including multiple imputations of the missing entries.

Related papers

• Generating Realistic Tabular Data with Large Language Models [49.03536886067729]
  Large language models (LLM) have been used for diverse tasks, but do not capture the correct correlation between the features and the target variable. We propose a LLM-based method with three important improvements to correctly capture the ground-truth feature-class correlation in the real data. Our experiments show that our method significantly outperforms 10 SOTA baselines on 20 datasets in downstream tasks.
  arXiv  Detail & Related papers  (2024-10-29T04:14:32Z)
• Towards a Theoretical Understanding of Memorization in Diffusion Models [76.85077961718875]
  Diffusion probabilistic models (DPMs) are being employed as mainstream models for Generative Artificial Intelligence (GenAI) We provide a theoretical understanding of memorization in both conditional and unconditional DPMs under the assumption of model convergence. We propose a novel data extraction method named textbfSurrogate condItional Data Extraction (SIDE) that leverages a time-dependent classifier trained on the generated data as a surrogate condition to extract training data from unconditional DPMs.
  arXiv  Detail & Related papers  (2024-10-03T13:17:06Z)
• Dataset Condensation with Latent Quantile Matching [5.466962214217334]
  Current distribution matching (DM) based DC methods learn a synthesized dataset by matching the mean of the latent embeddings between the synthetic and the real outliers. We propose Latent Quantile Matching (LQM) which matches the quantiles of the latent embeddings to minimize the goodness of fit test statistic between two distributions.
  arXiv  Detail & Related papers  (2024-06-14T09:20:44Z)
• Generative modeling of density regression through tree flows [3.0262553206264893]
  We propose a flow-based generative model tailored for the density regression task on tabular data. We introduce a training algorithm for fitting the tree-based transforms using a divide-and-conquer strategy. Our method consistently achieves comparable or superior performance at a fraction of the training and sampling budget.
  arXiv  Detail & Related papers  (2024-06-07T21:07:35Z)
• Minimally Supervised Learning using Topological Projections in Self-Organizing Maps [55.31182147885694]
  We introduce a semi-supervised learning approach based on topological projections in self-organizing maps (SOMs) Our proposed method first trains SOMs on unlabeled data and then a minimal number of available labeled data points are assigned to key best matching units (BMU) Our results indicate that the proposed minimally supervised model significantly outperforms traditional regression techniques.
  arXiv  Detail & Related papers  (2024-01-12T22:51:48Z)
• Learning from aggregated data with a maximum entropy model [73.63512438583375]
  We show how a new model, similar to a logistic regression, may be learned from aggregated data only by approximating the unobserved feature distribution with a maximum entropy hypothesis. We present empirical evidence on several public datasets that the model learned this way can achieve performances comparable to those of a logistic model trained with the full unaggregated data.
  arXiv  Detail & Related papers  (2022-10-05T09:17:27Z)
• Model-based Clustering with Missing Not At Random Data [0.8777702580252754]
  We propose model-based clustering algorithms designed to handle very general types of missing data, including MNAR data. Several MNAR models are discussed, for which the cause of the missingness can depend on both the values of the missing variable themselves and on the class membership. We focus on a specific MNAR model, called MNARz, for which the missingness only depends on the class membership.
  arXiv  Detail & Related papers  (2021-12-20T09:52:12Z)
• Mixing Deep Learning and Multiple Criteria Optimization: An Application to Distributed Learning with Multiple Datasets [0.0]
  Training phase is the most important stage during the machine learning process. We develop a multiple criteria optimization model in which each criterion measures the distance between the output associated with a specific input and its label. We propose a scalarization approach to implement this model and numerical experiments in digit classification using MNIST data.
  arXiv  Detail & Related papers  (2021-12-02T16:00:44Z)
• Autoregressive Score Matching [113.4502004812927]
  We propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariable log-conditionals (scores) For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
  arXiv  Detail & Related papers  (2020-10-24T07:01:24Z)
• BREEDS: Benchmarks for Subpopulation Shift [98.90314444545204]
  We develop a methodology for assessing the robustness of models to subpopulation shift. We leverage the class structure underlying existing datasets to control the data subpopulations that comprise the training and test distributions. Applying this methodology to the ImageNet dataset, we create a suite of subpopulation shift benchmarks of varying granularity.
  arXiv  Detail & Related papers  (2020-08-11T17:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.