Beyond the Fold: Quantifying Split-Level Noise and the Case for Leave-One-Dataset-Out AU Evaluation

• URL: http://arxiv.org/abs/2604.02162v1
• Date: Thu, 02 Apr 2026 15:28:55 GMT
• Title: Beyond the Fold: Quantifying Split-Level Noise and the Case for Leave-One-Dataset-Out AU Evaluation
• Abstract summary: We show that cross-validation itself introduces measurable variance.<n>On BP4D+, repeated 3-fold subject-exclusive splits produce an empirical noise floor of $pm 0.065$ in average F1.<n>We further evaluate cross-dataset robustness using a Leave-One-Dataset-Out protocol across five AU datasets.
• Score: 1.7159330469510958
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Subject-exclusive cross-validation is the standard evaluation protocol for facial Action Unit (AU) detection, yet reported improvements are often small. We show that cross-validation itself introduces measurable stochastic variance. On BP4D+, repeated 3-fold subject-exclusive splits produce an empirical noise floor of $\pm 0.065$ in average F1, with substantially larger variation for low-prevalence AUs. Operating-point metrics such as F1 fluctuate more than threshold-independent measures such as AUC, and model ranking can change under different fold assignments. We further evaluate cross-dataset robustness using a Leave-One-Dataset-Out (LODO) protocol across five AU datasets. LODO removes partition randomness and exposes domain-level instability that is not visible under single-dataset cross-validation. Together, these results suggest that gains often reported in cross-fold validation may fall within protocol variance. Leave-one-dataset-out cross-validation yields more stable and interpretable findings

Related papers

• Post-Hoc Trajectory-Risk Certification for Modular LLM-Based Security Agents [1.4915002678801434]
  We show that Bonferroni allocation is distribution-free but conservative under correlated errors.<n> coarse-to-fine label selection can create near-perfect measured correlation without learned dependence.<n>A modular certificate using per-stage certificates and a pairwise overlap bound yields a positive average gain of 0.6%.
  arXiv  Detail & Related papers  (2026-08-04T23:48:36Z)
• Gate AI: LLM Security Benchmark Evaluation Methodology and Results [0.0]
  We describe an evaluation harness that addresses per-dataset threshold tuning and undisclosed operating points.<n>The detector under evaluation is scored across 16 public benchmarks using 5-fold cross-validation.<n>For every external comparison, the detector's threshold is re-tuned to the competitor's published false-positive rate.
  arXiv  Detail & Related papers  (2026-06-01T23:29:58Z)
• Benchmarking Inductive Biases for Multivariate Time-Series Anomaly Detection with a Robust Multi-View Channel-Graph Detector [8.29991973809832]
  Ten family-representative detectors are evaluated on five datasets.<n>All methods share the same windowing, scoring, hardware, and metric protocols.<n>We introduce ours, an adaptive detector family combining a NOTEARS-constrained directed channel-graph view with optional patch-attention and temporal-association views.
  arXiv  Detail & Related papers  (2026-05-27T07:55:44Z)
• PAIR-CI: Calibrated Conditional Independence Testing for Causal Discovery with Incomplete Data [0.0]
  We introduce PAIR-CI, a nonparametric conditional independence (CI) test that restores calibration by integrating multiple imputation directly into the inferential procedure.<n>A provably consistent variance estimator jointly accounts for uncertainty arising from cross-validation and multiple imputation.
  arXiv  Detail & Related papers  (2026-05-06T12:34:37Z)
• Posterior Augmented Flow Matching [64.1559809786948]
  Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
  arXiv  Detail & Related papers  (2026-05-01T17:59:59Z)
• Towards One-for-All Anomaly Detection for Tabular Data [87.63505963517512]
  Tabular anomaly detection (TAD) is critical in many real-world applications.<n>We propose OFA-TAD, a framework that only requires one-time training on multiple source datasets.<n>Experiments on 34 datasets from 14 domains demonstrate that OFA-TAD achieves superior anomaly detection performance.
  arXiv  Detail & Related papers  (2026-03-15T14:42:05Z)
• A Sample Efficient Conditional Independence Test in the Presence of Discretization [54.047334792855345]
  Conditional Independence (CI) tests directly to discretized data can lead to incorrect conclusions.<n>Recent advancements have sought to infer the correct CI relationship between the latent variables through binarizing observed data.<n>Motivated by this, this paper introduces a sample-efficient CI test that does not rely on the binarization process.
  arXiv  Detail & Related papers  (2025-06-10T12:41:26Z)
• Mitigating covariate shift in non-colocated data with learned parameter priors [0.0]
  We present textitFragmentation-induced co-shift remediation ($FIcsR$), which minimizes an $f$-divergence between a fragment's covariate distribution and that of the standard cross-validation baseline. We run extensive classification experiments on multiple data classes, over $40$ datasets, and with data batched over multiple sequence lengths. The results are promising under all these conditions; with improved accuracy against batch and fold state-of-the-art by more than $5%$ and $10%$, respectively.
  arXiv  Detail & Related papers  (2024-11-10T15:48:29Z)
• Federated Nonparametric Hypothesis Testing with Differential Privacy Constraints: Optimal Rates and Adaptive Tests [5.3595271893779906]
  Federated learning has attracted significant recent attention due to its applicability across a wide range of settings where data is collected and analyzed across disparate locations. We study federated nonparametric goodness-of-fit testing in the white-noise-with-drift model under distributed differential privacy (DP) constraints.
  arXiv  Detail & Related papers  (2024-06-10T19:25:19Z)
• Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles [104.60508550106618]
  We propose DiffDiv, an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs)<n>We show that DPMs can generate images with novel feature combinations, even when trained on samples displaying correlated input features.<n>We show that DPM-guided diversification is sufficient to remove dependence on shortcut cues, without a need for additional supervised signals.
  arXiv  Detail & Related papers  (2023-11-23T15:47:33Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• Data thinning for convolution-closed distributions [2.299914829977005]
  We propose data thinning, an approach for splitting an observation into two or more independent parts that sum to the original observation. We show that data thinning can be used to validate the results of unsupervised learning approaches.
  arXiv  Detail & Related papers  (2023-01-18T02:47:41Z)
• Sparse Conditional Hidden Markov Model for Weakly Supervised Named Entity Recognition [68.68300358332156]
  We propose the sparse conditional hidden Markov model (Sparse-CHMM) to evaluate noisy labeling functions. Sparse-CHMM is optimized through unsupervised learning with a three-stage training pipeline. It achieves a 3.01 average F1 score improvement on five comprehensive datasets.
  arXiv  Detail & Related papers  (2022-05-27T20:47:30Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)
• Fast calculation of Gaussian Process multiple-fold cross-validation residuals and their covariances [0.6091702876917281]
  We generalize fast leave-one-out formulae to multiple-fold cross-validation. We highlight the covariance structure of cross-validation residuals in both Simple and Universal Kriging frameworks. Our results enable fast multiple-fold cross-validation and have direct consequences in model diagnostics.
  arXiv  Detail & Related papers  (2021-01-08T17:02:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.