GEM: Geometric Entropy Mixing for Optimal LLM Data Curation

• URL: http://arxiv.org/abs/2605.26121v1
• Date: Mon, 27 Apr 2026 06:42:28 GMT
• Title: GEM: Geometric Entropy Mixing for Optimal LLM Data Curation
• Abstract summary: We introduce GEM (Geometric Entropy Mixing), a framework reformulating data curation as a variational problem.<n>By decoupling the generative variational problem with a mixing-balance prior, GEM effectively counteracts the cluster collapse.
• Score: 5.2826294472260935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM pre-training efficacy increasingly depends on data composition rather than sheer volume. Yet, optimal mixing is hindered by categorization flaws: human taxonomies suffer from ontological misalignment, and Euclidean clustering fails to address embedding anisotropy. We introduce GEM (Geometric Entropy Mixing), a framework reformulating data curation as a variational problem on the hypersphere augmented with a mixing-balance regularizer. By decoupling the generative prior and optimizing the objective via a provable MM (Minorize-Maximize) algorithm, GEM effectively counteracts the cluster collapse to discover balanced semantic structures invisible to Euclidean heuristics. We employ teacher-student distillation to scale this geometric fidelity to web-scale corpora and introduce the Geometric Influence Score (GIS) for interpretable taxonomy generation. Experiments with 1.1B-parameter models demonstrate that GEM establishes a new state-of-the-art when integrated into mixing strategies like DoReMi and RegMix, improving average downstream accuracy by up to 1.2% and offering a robust coordinate system for predictable data mixing.

Related papers

• Joint Model and Data Sparsification via the Marginal Likelihood [53.29070892356214]
  We propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification.<n>This symmetric pruning of model and data offers a natural extension that preserves conjugacy.<n> Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.
  arXiv  Detail & Related papers  (2026-05-28T13:26:53Z)
• From Simple to Complex: Curriculum-Guided Physics-Informed Neural Networks via Gaussian Mixture Models [13.535408662482434]
  Physics-informed neural networks (PINNs) offer a mesh-free framework for solving partial differential equations (PDEs)<n>We propose the CurriculumGuided Mixture Physics-Informed Neural Network (CGMPINN), which integrates mixture modeling with dynamic curriculum learning.
  arXiv  Detail & Related papers  (2026-05-19T02:20:13Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Physically Interpretable Representation Learning with Gaussian Mixture Variational AutoEncoder (GM-VAE) [37.18249990338269]
  We propose a Variational Autoencoder (GM-VAE) framework designed to extract, physically interpretable representations from high-dimensional scientific data.<n>Unlike conventional VAEs that jointly optimize reconstruction and clustering, our method utilizes a block-coordinate descent strategy.<n>To objectively evaluate the learned representations, we introduce a metric based on graph-Laplacian smoothness, which measures the coherence of physical instability across the latent manifold.
  arXiv  Detail & Related papers  (2025-11-26T20:04:38Z)
• Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency without Model Sweeps [41.371172458797524]
  Non-identifiability of gating parameters up to common translations, intrinsic gate-expert interactions, and tight numerator-denominator coupling are addressed.<n>For model selection, we adapt dendrogram-guided SGMoE, yielding a consistent, sweep-free selector of the number of experts that attains optimal parameter rates.<n>On a dataset of drought-identifiable maize traits, our dendrogram-guided SGMoE selects two experts, exposes a clear mixing hierarchy, stabilizes the likelihood early, and yields interpretable genotype-phenotype maps.
  arXiv  Detail & Related papers  (2025-10-14T17:23:44Z)
• Interaction-Aware Gaussian Weighting for Clustered Federated Learning [58.92159838586751]
  Federated Learning (FL) emerged as a decentralized paradigm to train models while preserving privacy.<n>We propose a novel clustered FL method, FedGWC (Federated Gaussian Weighting Clustering), which groups clients based on their data distribution.<n>Our experiments on benchmark datasets show that FedGWC outperforms existing FL algorithms in cluster quality and classification accuracy.
  arXiv  Detail & Related papers  (2025-02-05T16:33:36Z)
• A Fresh Look at Generalized Category Discovery through Non-negative Matrix Factorization [83.12938977698988]
  Generalized Category Discovery (GCD) aims to classify both base and novel images using labeled base data. Current approaches inadequately address the intrinsic optimization of the co-occurrence matrix $barA$ based on cosine similarity. We propose a Non-Negative Generalized Category Discovery (NN-GCD) framework to address these deficiencies.
  arXiv  Detail & Related papers  (2024-10-29T07:24:11Z)
• Adaptive Fuzzy C-Means with Graph Embedding [84.47075244116782]
  Fuzzy clustering algorithms can be roughly categorized into two main groups: Fuzzy C-Means (FCM) based methods and mixture model based methods. We propose a novel FCM based clustering model that is capable of automatically learning an appropriate membership degree hyper- parameter value.
  arXiv  Detail & Related papers  (2024-05-22T08:15:50Z)
• Towards the Theory of Unsupervised Federated Learning: Non-asymptotic Analysis of Federated EM Algorithms [13.857921574409362]
  We introduce a federated gradient EM algorithm (FedGrEM) designed for the unsupervised learning of mixture models. We present a comprehensive finite-sample theory that holds for general mixture models. We then apply this general theory on specific statistical models to characterize the explicit estimation error of model parameters and mixture proportions.
  arXiv  Detail & Related papers  (2023-10-23T19:53:36Z)
• SIGMA: Scale-Invariant Global Sparse Shape Matching [50.385414715675076]
  We propose a novel mixed-integer programming (MIP) formulation for generating precise sparse correspondences for non-rigid shapes. We show state-of-the-art results for sparse non-rigid matching on several challenging 3D datasets.
  arXiv  Detail & Related papers  (2023-08-16T14:25:30Z)
• A distribution-free mixed-integer optimization approach to hierarchical modelling of clustered and longitudinal data [0.0]
  We introduce an innovative algorithm that evaluates cluster effects for new data points, thereby increasing the robustness and precision of this model. The inferential and predictive efficacy of this approach is further illustrated through its application in student scoring and protein expression.
  arXiv  Detail & Related papers  (2023-02-06T23:34:51Z)
• Likelihood Adjusted Semidefinite Programs for Clustering Heterogeneous Data [16.153709556346417]
  Clustering is a widely deployed learning tool. iLA-SDP is less sensitive than EM to and more stable on high-dimensional data.
  arXiv  Detail & Related papers  (2022-09-29T21:03:13Z)
• Hierarchical mixtures of Gaussians for combined dimensionality reduction and clustering [6.635611625764804]
  We introduce hierarchical mixtures of Gaussians (HMoGs), which unify dimensionality reduction and clustering into a single model.<n>HMoGs provide closed-form expressions for the model likelihood, exact inference over latent states and cluster membership, and exact algorithms for maximum-likelihood optimization.<n>We demonstrate HMoGs on synthetic experiments and MNIST, and show how joint optimization of dimensionality reduction and clustering facilitates increased model performance.
  arXiv  Detail & Related papers  (2022-06-10T02:03:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.