e-Fold Cross-Validation for Recommender-System Evaluation

• URL: http://arxiv.org/abs/2412.01011v1
• Date: Mon, 02 Dec 2024 00:05:20 GMT
• Title: e-Fold Cross-Validation for Recommender-System Evaluation
• Authors: Moritz Baumgart, Lukas Wegmeth, Tobias Vente, Joeran Beel,
• Abstract summary: e-fold cross validation aims to minimize the number of folds to achieve a reduction in power usage.<n>On average e-fold cross validation only needed 41.5% of the energy that 10-fold cross validation would need, while it's results only differed by 1.81%.
• Score: 0.09374652839580183
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: To combat the rising energy consumption of recommender systems we implement a novel alternative for k-fold cross validation. This alternative, named e-fold cross validation, aims to minimize the number of folds to achieve a reduction in power usage while keeping the reliability and robustness of the test results high. We tested our method on 5 recommender system algorithms across 6 datasets and compared it with 10-fold cross validation. On average e-fold cross validation only needed 41.5% of the energy that 10-fold cross validation would need, while it's results only differed by 1.81%. We conclude that e-fold cross validation is a promising approach that has the potential to be an energy efficient but still reliable alternative to k-fold cross validation.

Related papers

• Towards Anytime-Valid Statistical Watermarking [63.02116925616554]
  We develop the first e-value-based watermarking framework, Anchored E-Watermarking, that unifies optimal sampling with anytime-valid inference.<n>Our framework can significantly enhance sample efficiency, reducing the average token budget required for detection by 13-15% relative to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-02-19T18:32:26Z)
• PRIME: A Process-Outcome Alignment Benchmark for Verifiable Reasoning in Mathematics and Engineering [71.15346406323827]
  We introduce PRIME, a benchmark for evaluating verifiers on Process-Outcome Alignment verification.<n>We find that current verifiers frequently fail to detect derivation flaws.<n>We propose a process-aware RLVR training paradigm utilizing verifiers selected via PRIME.
  arXiv  Detail & Related papers  (2026-02-12T04:45:01Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• Stability Regularized Cross-Validation [5.156484100374059]
  We revisit the problem of ensuring strong test-set performance via cross-validation.<n>Motivated by the generalization theory literature, we propose a nested k-fold cross-validation scheme.<n>We benchmark our procedure on a suite of 13 real-world UCI datasets.
  arXiv  Detail & Related papers  (2025-05-11T10:06:59Z)
• Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judge [90.8674158031845]
  We propose Crowd-based Comparative Evaluation, which introduces additional crowd responses to compare with the candidate responses. This process effectively guides LLM-as-a-Judge to provide a more detailed chain-of-thought (CoT) judgment. Our method produces higher-quality CoTs that facilitate judge distillation and exhibit superior performance in rejection sampling.
  arXiv  Detail & Related papers  (2025-02-18T03:31:06Z)
• Test Input Validation for Vision-based DL Systems: An Active Learning Approach [3.760715803298828]
  Testing deep learning (DL) systems requires extensive and diverse, yet valid, test inputs. We propose a test input validation approach for vision-based DL systems.
  arXiv  Detail & Related papers  (2025-01-03T02:50:43Z)
• From Theory to Practice: Implementing and Evaluating e-Fold Cross-Validation [0.10241134756773229]
  e-fold cross-validation is an energy-efficient alternative to k-fold cross-validation. It requires 4 fewer folds than 10-fold cross-validation, reducing evaluation time, computational resources, and energy use by about 40%. E-fold cross-validation offers a reliable and efficient alternative to k-fold, reducing computational costs while maintaining comparable accuracy.
  arXiv  Detail & Related papers  (2024-10-12T09:56:28Z)
• On Speeding Up Language Model Evaluation [48.51924035873411]
  Development of prompt-based methods with Large Language Models (LLMs) requires making numerous decisions. We propose a novel method to address this challenge. We show that it can identify the top-performing method using only 5-15% of the typically needed resources.
  arXiv  Detail & Related papers  (2024-07-08T17:48:42Z)
• Don't Waste Your Time: Early Stopping Cross-Validation [41.092016771160566]
  Cross-validation drastically increases the computational cost of validating a single configuration. Our study shows that even a simple-to-understand and easy-to-implement method consistently allows model selection to converge faster.
  arXiv  Detail & Related papers  (2024-05-06T11:51:09Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• Delving into Probabilistic Uncertainty for Unsupervised Domain Adaptive Person Re-Identification [54.174146346387204]
  We propose an approach named probabilistic uncertainty guided progressive label refinery (P$2$LR) for domain adaptive person re-identification. A quantitative criterion is established to measure the uncertainty of pseudo labels and facilitate the network training. Our method outperforms the baseline by 6.5% mAP on the Duke2Market task, while surpassing the state-of-the-art method by 2.5% mAP on the Market2MSMT task.
  arXiv  Detail & Related papers  (2021-12-28T07:40:12Z)
• End-to-End Semi-Supervised Object Detection with Soft Teacher [63.26266730447914]
  This paper presents an end-to-end semi-supervised object detection approach, in contrast to previous more complex multi-stage methods. The proposed approach outperforms previous methods by a large margin under various labeling ratios. On the state-of-the-art Swin Transformer-based object detector, it can still significantly improve the detection accuracy by +1.5 mAP.
  arXiv  Detail & Related papers  (2021-06-16T17:59:30Z)
• Cross-validation Confidence Intervals for Test Error [83.67415139421448]
  This work develops central limit theorems for crossvalidation and consistent estimators of its variance under weak stability conditions on the learning algorithm. Results are the first of their kind for the popular choice of leave-one-out cross-validation.
  arXiv  Detail & Related papers  (2020-07-24T17:40:06Z)
• Second-Order Provable Defenses against Adversarial Attacks [63.34032156196848]
  We show that if the eigenvalues of the network are bounded, we can compute a certificate in the $l$ norm efficiently using convex optimization. We achieve certified accuracy of 5.78%, and 44.96%, and 43.19% on 2,59% and 4BP-based methods respectively.
  arXiv  Detail & Related papers  (2020-06-01T05:55:18Z)
• Bootstrap Bias Corrected Cross Validation applied to Super Learning [0.3670422696827526]
  Super learner algorithm can be applied to combine results of multiple base learners to improve quality of predictions. Tests were performed on artificial data sets of diverse size and on seven real, biomedical data sets. The resampling method, called Bootstrap Bias Correction, proved to be a reasonably precise and very cost-efficient alternative for nested cross validation.
  arXiv  Detail & Related papers  (2020-03-18T17:12:42Z)
• Towards new cross-validation-based estimators for Gaussian process regression: efficient adjoint computation of gradients [0.0]
  We suggest using new cross-validation criteria derived from the literature of scoring rules. We also provide an efficient method for computing the gradient of a cross-validation criterion.
  arXiv  Detail & Related papers  (2020-02-26T14:50:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.