Rashomon effect in Educational Research: Why More is Better Than One for Measuring the Importance of the Variables?

• URL: http://arxiv.org/abs/2412.12115v1
• Date: Mon, 02 Dec 2024 14:05:36 GMT
• Title: Rashomon effect in Educational Research: Why More is Better Than One for Measuring the Importance of the Variables?
• Authors: Jakub Kuzilek, Mustafa Çavuş,
• Abstract summary: The study uses the Rashomon set of simple-yet-accurate models trained using decision trees, random forests, light GBM, and XGBoost algorithms. We found that the Rashomon set improves the predictive accuracy by 2-6%. Key demographic variables imd_band and highest_education were identified as vital, but their importance varied across courses.
• Score: 0.0
• License:
• Abstract: This study explores how the Rashomon effect influences variable importance in the context of student demographics used for academic outcomes prediction. Our research follows the way machine learning algorithms are employed in Educational Data Mining, focusing on highlighting the so-called Rashomon effect. The study uses the Rashomon set of simple-yet-accurate models trained using decision trees, random forests, light GBM, and XGBoost algorithms with the Open University Learning Analytics Dataset. We found that the Rashomon set improves the predictive accuracy by 2-6%. Variable importance analysis revealed more consistent and reliable results for binary classification than multiclass classification, highlighting the complexity of predicting multiple outcomes. Key demographic variables imd_band and highest_education were identified as vital, but their importance varied across courses, especially in course DDD. These findings underscore the importance of model choice and the need for caution in generalizing results, as different models can lead to different variable importance rankings. The codes for reproducing the experiments are available in the repository: https://anonymous.4open.science/r/JEDM_paper-DE9D.

Related papers

• Exploring the Efficacy of Meta-Learning: Unveiling Superior Data Diversity Utilization of MAML Over Pre-training [1.3980986259786223]
  We show that dataset diversity can impact the performance of vision models. Our study shows positive correlations between test set accuracy and data diversity. These findings support our hypothesis and demonstrate a promising way for a deeper exploration of how formal data diversity influences model performance.
  arXiv  Detail & Related papers  (2025-01-15T00:56:59Z)
• Unraveling the Key Components of OOD Generalization via Diversification [20.064947636966078]
  Supervised learning datasets may contain multiple cues that explain the training set equally well. Many of them can be spurious, i.e., lose their predictive power under a distribution shift. Recently developed "diversification" methods approach this problem by finding multiple diverse hypotheses.
  arXiv  Detail & Related papers  (2023-12-26T19:47:53Z)
• A Path to Simpler Models Starts With Noise [17.36067410506525]
  The Rashomon set is the set of models that perform approximately equally well on a given dataset. An open question is why Rashomon ratios often tend to be large. We show that noisier datasets lead to larger Rashomon ratios through the way that practitioners train models.
  arXiv  Detail & Related papers  (2023-10-30T16:52:57Z)
• The Rashomon Importance Distribution: Getting RID of Unstable, Single Model-based Variable Importance [16.641794438414745]
  Quantifying variable importance is essential for answering high-stakes questions in fields like genetics, public policy, and medicine. We propose a new variable importance framework that quantifies the importance of a variable across the set of all good models and is stable across the data distribution. Our framework is extremely flexible and can be integrated with most existing model classes and global variable importance metrics.
  arXiv  Detail & Related papers  (2023-09-24T23:09:48Z)
• The XAISuite framework and the implications of explanatory system dissonance [0.0]
  This paper compares two explanatory systems, SHAP and LIME, based on the correlation of their respective importance scores. The magnitude of importance is not significant in explanation consistency. The similarity between SHAP and LIME importance scores cannot predict model accuracy.
  arXiv  Detail & Related papers  (2023-04-15T04:40:03Z)
• ASPEST: Bridging the Gap Between Active Learning and Selective Prediction [56.001808843574395]
  Selective prediction aims to learn a reliable model that abstains from making predictions when uncertain. Active learning aims to lower the overall labeling effort, and hence human dependence, by querying the most informative examples. In this work, we introduce a new learning paradigm, active selective prediction, which aims to query more informative samples from the shifted target domain.
  arXiv  Detail & Related papers  (2023-04-07T23:51:07Z)
• Do Deep Neural Networks Always Perform Better When Eating More Data? [82.6459747000664]
  We design experiments from Identically Independent Distribution(IID) and Out of Distribution(OOD) Under IID condition, the amount of information determines the effectivity of each sample, the contribution of samples and difference between classes determine the amount of class information. Under OOD condition, the cross-domain degree of samples determine the contributions, and the bias-fitting caused by irrelevant elements is a significant factor of cross-domain.
  arXiv  Detail & Related papers  (2022-05-30T15:40:33Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• General Greedy De-bias Learning [163.65789778416172]
  We propose a General Greedy De-bias learning framework (GGD), which greedily trains the biased models and the base model like gradient descent in functional space. GGD can learn a more robust base model under the settings of both task-specific biased models with prior knowledge and self-ensemble biased model without prior knowledge.
  arXiv  Detail & Related papers  (2021-12-20T14:47:32Z)
• X-model: Improving Data Efficiency in Deep Learning with A Minimax Model [78.55482897452417]
  We aim at improving data efficiency for both classification and regression setups in deep learning. To take the power of both worlds, we propose a novel X-model. X-model plays a minimax game between the feature extractor and task-specific heads.
  arXiv  Detail & Related papers  (2021-10-09T13:56:48Z)
• Can Active Learning Preemptively Mitigate Fairness Issues? [66.84854430781097]
  dataset bias is one of the prevailing causes of unfairness in machine learning. We study whether models trained with uncertainty-based ALs are fairer in their decisions with respect to a protected class. We also explore the interaction of algorithmic fairness methods such as gradient reversal (GRAD) and BALD.
  arXiv  Detail & Related papers  (2021-04-14T14:20:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.