Enhancing Automated Machine Learning via Homogeneous Train-Test Splitting Methods

• URL: http://arxiv.org/abs/2607.26625v1
• Date: Wed, 29 Jul 2026 08:53:11 GMT
• Title: Enhancing Automated Machine Learning via Homogeneous Train-Test Splitting Methods
• Abstract summary: This paper investigates the role of statistical similarity in train-test splitting and its consequences for AutoML model evaluation.<n>Similarity is assessed using chi-square, Kolmogorov-Smirnov, and Maximum Mean Discrepancy (MMD) tests.<n>The proposed Optimised-Distribution method treats similarity as an explicit optimisation objective and achieves the highest mean MMD similarity, 89.0%, across all strategies evaluated.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate model evaluation in machine learning depends critically on how datasets are split into training and testing subsets. Standard random splitting assumes that both partitions share the same underlying distribution, an assumption often violated in datasets with class imbalance, natural clustering, or spatial autocorrelation. This paper investigates the role of statistical similarity in train-test splitting and its consequences for AutoML model evaluation. Five established strategies are compared across fifteen UCI benchmark datasets: random splitting, stratified sampling, Kennard-Stone, Duplex, and SPXY. Similarity is assessed using chi-square, Kolmogorov-Smirnov, and Maximum Mean Discrepancy (MMD) tests. Geometry-based methods consistently produce near-zero MMD scores, introducing instability into downstream performance estimates. The proposed Optimised-Distribution method treats similarity as an explicit optimisation objective and achieves the highest mean MMD similarity, 89.0%, across all strategies evaluated.

Related papers

• Beyond the Performance Illusion: Structure-Aware Stratified Partitioning and Curriculum Distributionally Robust Optimization for Spatially Correlated Domains [1.8595194352358455]
  We propose a unified evaluation and training framework for spatially correlated data.<n>We show that this combination yields consistently improved generalization, more reliable confidence calibration, and exposes modes failure that remain hidden under conventional random-split evaluation.
  arXiv  Detail & Related papers  (2026-07-02T11:32:38Z)
• Revisiting OmniAnomaly for Anomaly Detection: performance metrics and comparison with PCA-based models [0.08155575318208629]
  This work revisits OmniAnomaly, a widely used recurrent model for MTSAD, and compares it with a simple linear baseline based on Principal Component Analysis.<n>Both methods are evaluated under thresholding and evaluation procedures, with experiments repeated across 100 runs for each of the 28 machines in the dataset.<n>The results show large variability across machines and show that can achieve performance comparable to OmniAnomaly, and even outperform it when point-adjustment is not applied.
  arXiv  Detail & Related papers  (2026-03-19T14:54:38Z)
• Clustering Approaches for Mixed-Type Data: A Comparative Study [0.0]
  Clustering mixed-type data is a challenge, as few existing approaches are suited for this task.<n>This study presents the state-of-the-art of these approaches and compares them using various simulation models.<n>In our experiments KAMILA, LCM, and k-prototypes exhibited the best performance, with respect to the adjusted rand index (ARI)
  arXiv  Detail & Related papers  (2025-11-24T22:18:23Z)
• Nonparametric Data Attribution for Diffusion Models [57.820618036556084]
  Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs.<n>We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images.
  arXiv  Detail & Related papers  (2025-10-16T03:37:16Z)
• The Lie of the Average: How Class Incremental Learning Evaluation Deceives You? [48.83567710215299]
  Class Incremental Learning (CIL) requires models to continuously learn new classes without forgetting previously learned ones.<n>We argue that a robust CIL evaluation protocol should accurately characterize and estimate the entire performance distribution.<n>We propose EDGE, an evaluation protocol that adaptively identifies and samples extreme class sequences using inter-task similarity.
  arXiv  Detail & Related papers  (2025-09-26T17:00:15Z)
• Multiply Robust Estimation for Local Distribution Shifts with Multiple Domains [9.429772474335122]
  We focus on scenarios where data distributions vary across multiple segments of the entire population. We propose a two-stage multiply robust estimation method to improve model performance on each individual segment. Our method is designed to be implemented with commonly used off-the-shelf machine learning models.
  arXiv  Detail & Related papers  (2024-02-21T22:01:10Z)
• A structured regression approach for evaluating model performance across intersectional subgroups [53.91682617836498]
  Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups. We introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups.
  arXiv  Detail & Related papers  (2024-01-26T14:21:45Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Similarity Based Stratified Splitting: an approach to train better classifiers [0.0]
  We propose a Similarity-Based Stratified Splitting technique, which uses both the output and input space information to split the data. We evaluate our proposal in twenty-two benchmark datasets with classifiers such as Multi-Layer Perceptron, Support Vector Machine, Random Forest and K-Nearest Neighbors.
  arXiv  Detail & Related papers  (2020-10-13T01:07:48Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)
• Clustering Binary Data by Application of Combinatorial Optimization Heuristics [52.77024349608834]
  We study clustering methods for binary data, first defining aggregation criteria that measure the compactness of clusters. Five new and original methods are introduced, using neighborhoods and population behavior optimization metaheuristics. From a set of 16 data tables generated by a quasi-Monte Carlo experiment, a comparison is performed for one of the aggregations using L1 dissimilarity, with hierarchical clustering, and a version of k-means: partitioning around medoids or PAM.
  arXiv  Detail & Related papers  (2020-01-06T23:33:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.