Better Peer Grading through Bayesian Inference

• URL: http://arxiv.org/abs/2209.01242v1
• Date: Fri, 2 Sep 2022 19:10:53 GMT
• Title: Better Peer Grading through Bayesian Inference
• Authors: Hedayat Zarkoob and Greg d'Eon and Lena Podina and Kevin Leyton-Brown
• Abstract summary: Peer grading systems aggregate noisy reports from multiple students to approximate a true grade as closely as possible. This paper extends the state of the art in three key ways: (1) recognizing that students can behave strategically; (2) appropriately handling censored data that arises from discrete-valued grading rubrics; and (3) using mixed integer programming to improve the interpretability of the grades assigned to students.
• Score: 13.113568233352986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Peer grading systems aggregate noisy reports from multiple students to approximate a true grade as closely as possible. Most current systems either take the mean or median of reported grades; others aim to estimate students' grading accuracy under a probabilistic model. This paper extends the state of the art in the latter approach in three key ways: (1) recognizing that students can behave strategically (e.g., reporting grades close to the class average without doing the work); (2) appropriately handling censored data that arises from discrete-valued grading rubrics; and (3) using mixed integer programming to improve the interpretability of the grades assigned to students. We show how to make Bayesian inference practical in this model and evaluate our approach on both synthetic and real-world data obtained by using our implemented system in four large classes. These extensive experiments show that grade aggregation using our model accurately estimates true grades, students' likelihood of submitting uninformative grades, and the variation in their inherent grading error; we also characterize our models' robustness.

Related papers

• A Fair Post-Processing Method based on the MADD Metric for Predictive Student Models [1.055551340663609]
  A new metric has been developed to evaluate algorithmic fairness in predictive student models. In this paper, we develop a post-processing method that aims at improving the fairness while preserving the accuracy of relevant predictive models' results. We experiment with our approach on the task of predicting student success in an online course, using both simulated and real-world educational data.
  arXiv  Detail & Related papers  (2024-07-07T14:53:41Z)
• Modeling the Uncertainty with Maximum Discrepant Students for Semi-supervised 2D Pose Estimation [57.17120203327993]
  We propose a framework to estimate the quality of pseudo-labels in semi-supervised pose estimation tasks. Our method improves the performance of semi-supervised pose estimation on three datasets.
  arXiv  Detail & Related papers  (2023-11-03T08:11:06Z)
• Cognition-Mode Aware Variational Representation Learning Framework for Knowledge Tracing [3.3036318543432417]
  The Knowledge Tracing task plays a crucial role in personalized learning. The KT task suffers from data sparsity, which makes it challenging to learn robust representations for students with few practice records. We propose a Cognition-Mode Aware Variational Representation Learning Framework (CMVF) that can be directly applied to existing KT methods.
  arXiv  Detail & Related papers  (2023-09-03T13:51:06Z)
• Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction [50.62245481416744]
  We present the first benchmark that simulates the evaluation of open information extraction models in the real world. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
  arXiv  Detail & Related papers  (2023-05-23T12:05:09Z)
• Robust Outlier Rejection for 3D Registration with Variational Bayes [70.98659381852787]
  We develop a novel variational non-local network-based outlier rejection framework for robust alignment. We propose a voting-based inlier searching strategy to cluster the high-quality hypothetical inliers for transformation estimation.
  arXiv  Detail & Related papers  (2023-04-04T03:48:56Z)
• Protected Attributes Tell Us Who, Behavior Tells Us How: A Comparison of Demographic and Behavioral Oversampling for Fair Student Success Modeling [6.58879009604603]
  We analyze the fairness of models which use behavioral data to identify at-risk students and suggest two novel pre-processing approaches for bias mitigation. Based on the concept of intersectionality, the first approach involves intelligent oversampling on combinations of demographic attributes. The second approach does not require any knowledge of demographic attributes and is based on the assumption that such attributes are a (noisy) proxy for student behavior.
  arXiv  Detail & Related papers  (2022-12-20T11:09:11Z)
• Efficient Action Recognition Using Confidence Distillation [9.028144245738247]
  We propose a confidence distillation framework to teach a representation of uncertainty of the teacher to the student sampler. We conduct extensive experiments on three action recognition datasets and demonstrate that our framework achieves significant improvements in action recognition accuracy (up to 20%) and computational efficiency (more than 40%)
  arXiv  Detail & Related papers  (2021-09-05T18:25:49Z)
• Integrating Semantics and Neighborhood Information with Graph-Driven Generative Models for Document Retrieval [51.823187647843945]
  In this paper, we encode the neighborhood information with a graph-induced Gaussian distribution, and propose to integrate the two types of information with a graph-driven generative model. Under the approximation, we prove that the training objective can be decomposed into terms involving only singleton or pairwise documents, enabling the model to be trained as efficiently as uncorrelated ones.
  arXiv  Detail & Related papers  (2021-05-27T11:29:03Z)
• Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
  Learning from implicit feedback is challenging because of the difficult nature of the one-class problem. Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem. We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
  arXiv  Detail & Related papers  (2021-05-11T03:38:16Z)
• VaB-AL: Incorporating Class Imbalance and Difficulty with Variational Bayes for Active Learning [38.33920705605981]
  We propose a method that can naturally incorporate class imbalance into the Active Learning framework. We show that our method can be applied to tasks classification on multiple different datasets.
  arXiv  Detail & Related papers  (2020-03-25T07:34:06Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.