Demographic Metadata as Construct-Irrelevant Noise in DistilBERT-Based Automated Essay Scoring

• URL: http://arxiv.org/abs/2606.21066v1
• Date: Fri, 19 Jun 2026 03:32:27 GMT
• Title: Demographic Metadata as Construct-Irrelevant Noise in DistilBERT-Based Automated Essay Scoring
• Abstract summary: This study investigates the impact of a specific multimodal fusion strategy on the predictive accuracy, training convergence, and score parity of a DistilBERT-based AES model.<n> Evaluated via 10-fold cross-validation, the findings reveal that the early fusion of demographic metadata and the input significantly degrades the model's overall accuracy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated Essay Scoring (AES) systems are increasingly used to support teachers in managing grading workloads and to provide a supplementary rater in large-scale assessments. While human grading is frequently influenced by students' demographic characteristics, the efficacy of different strategies for integrating demographic metadata with textual input used to train AES models remains underexplored. This study investigates the impact of a specific multimodal fusion strategy - naive metadata concatenation - on the predictive accuracy, training convergence, and score parity of a DistilBERT-based AES model. A comparative analysis was conducted using the ASAP 2.0 dataset to evaluate a baseline model against an experimental model trained with input that concatenates tokenised text and demographic metadata using a naive multimodal fusion strategy. Evaluated via 10-fold cross-validation, the findings reveal that the early fusion of demographic metadata and the input significantly degrades the model's overall predictive accuracy. The baseline model achieved a Quadratic Weighted Kappa (QWK) of 0.727, which dropped to 0.656 upon integrating metadata. Furthermore, the experimental model exhibited higher validation loss (1.29) compared to the baseline model (1.25). The experimental model also displayed exacerbated scoring bias, reducing score parity instances from 15 to 12 out of 19 tests.

Related papers

• Meta-Ensemble Learning with Diverse Data Splits for Improved Respiratory Sound Classification [4.790443834378345]
  Training reliable respiratory sound classification models remains challenging due to the limited size and subject diversity of datasets.<n>We investigate a meta-ensemble learning methodology that enhances prediction diversity by training base models on diverse data splits.<n>Our approach achieves new state-of-the-art performance on the ICBHI benchmark, reaching a Score of 66.49%.
  arXiv  Detail & Related papers  (2026-04-27T06:38:57Z)
• Investigating the Impact of Histopathological Foundation Models on Regressive Prediction of Homologous Recombination Deficiency [52.50039435394964]
  We systematically evaluate foundation models for regression-based tasks.<n>We extract patch-level features from whole slide images (WSI) using five state-of-the-art foundation models.<n>Models are trained to predict continuous HRD scores based on these extracted features across breast, endometrial, and lung cancer cohorts.
  arXiv  Detail & Related papers  (2026-01-29T14:06:50Z)
• The GT-Score: A Robust Objective Function for Reducing Overfitting in Data-Driven Trading Strategies [51.56484100374058]
  GT-Score is a composite objective function that integrates performance, statistical significance, consistency, and downside risk.<n>In walk-forward validation, GT-Score improves the generalization ratio by 98% relative to baseline objective functions.<n>These results suggest that embedding an anti-overfitting structure into the objective can improve the reliability of backtests in quantitative research.
  arXiv  Detail & Related papers  (2026-01-22T05:16:47Z)
• Fine-tuning Pre-trained Audio Models for COVID-19 Detection: A Technical Report [0.9431368999053936]
  This report investigates the performance of pre-trained audio models on COVID-19 detection tasks using established benchmark datasets.<n>We implement a strict demographic stratification by age and gender to prevent models from exploiting spurious correlations between demographic characteristics and COVID-19 status.
  arXiv  Detail & Related papers  (2025-11-18T21:54:20Z)
• Model-agnostic Mitigation Strategies of Data Imbalance for Regression [0.0]
  Data imbalance persists as a pervasive challenge in regression tasks, introducing bias in model performance and undermining predictive reliability.<n>We present advanced mitigation techniques, which build upon and improve existing sampling methods.<n>We demonstrate that constructing an ensemble of models -- one trained with imbalance mitigation and another without -- can significantly reduce these negative effects.
  arXiv  Detail & Related papers  (2025-06-02T09:46:08Z)
• Evaluating Sample Utility for Efficient Data Selection by Mimicking Model Weights [11.237906163959908]
  Multimodal models are trained on large-scale web-crawled datasets.<n>These datasets often contain noise, bias, and irrelevant information.<n>We propose an efficient, model-based approach using the Mimic Score.
  arXiv  Detail & Related papers  (2025-01-12T04:28:14Z)
• MITA: Bridging the Gap between Model and Data for Test-time Adaptation [68.62509948690698]
  Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. We propose Meet-In-The-Middle based MITA, which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions.
  arXiv  Detail & Related papers  (2024-10-12T07:02:33Z)
• TRIAGE: Characterizing and auditing training data for improved regression [80.11415390605215]
  We introduce TRIAGE, a novel data characterization framework tailored to regression tasks and compatible with a broad class of regressors. TRIAGE utilizes conformal predictive distributions to provide a model-agnostic scoring method, the TRIAGE score. We show that TRIAGE's characterization is consistent and highlight its utility to improve performance via data sculpting/filtering, in multiple regression settings.
  arXiv  Detail & Related papers  (2023-10-29T10:31:59Z)
• RoBLEURT Submission for the WMT2021 Metrics Task [72.26898579202076]
  We present our submission to the Shared Metrics Task: RoBLEURT. Our model reaches state-of-the-art correlations with the WMT 2020 human annotations upon 8 out of 10 to-English language pairs.
  arXiv  Detail & Related papers  (2022-04-28T08:49:40Z)
• Using Explainable Boosting Machine to Compare Idiographic and Nomothetic Approaches for Ecological Momentary Assessment Data [2.0824228840987447]
  This paper explores the use of non-linear interpretable machine learning (ML) models in classification problems. Various ensembles of trees are compared to linear models using imbalanced synthetic and real-world datasets. In one of the two real-world datasets, knowledge distillation method achieves improved AUC scores.
  arXiv  Detail & Related papers  (2022-04-04T17:56:37Z)
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data [62.29031007761901]
  This paper proposes a novel contrastive regularized clinical classification model. We introduce two unique positive sampling strategies specifically tailored for EHR data. Our framework yields highly competitive experimental results in predicting the mortality risk on real-world COVID-19 EHR data.
  arXiv  Detail & Related papers  (2021-04-07T06:02:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.