Fair Grading Algorithms for Randomized Exams
- URL: http://arxiv.org/abs/2304.06254v1
- Date: Thu, 13 Apr 2023 04:21:37 GMT
- Title: Fair Grading Algorithms for Randomized Exams
- Authors: Jiale Chen, Jason Hartline and Onno Zoeter
- Abstract summary: In a randomized exam, each student is asked a small number of random questions from a large question bank.
The predominant grading rule is simple averaging, i.e., calculating grades by averaging scores on the questions each student is asked, which is fair ex-ante, over the randomized questions, but not fair ex-post, on the realized questions.
The fair grading problem is to estimate the average grade of each student on the full question bank.
- Score: 0.5801044612920815
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper studies grading algorithms for randomized exams. In a randomized
exam, each student is asked a small number of random questions from a large
question bank. The predominant grading rule is simple averaging, i.e.,
calculating grades by averaging scores on the questions each student is asked,
which is fair ex-ante, over the randomized questions, but not fair ex-post, on
the realized questions. The fair grading problem is to estimate the average
grade of each student on the full question bank. The maximum-likelihood
estimator for the Bradley-Terry-Luce model on the bipartite student-question
graph is shown to be consistent with high probability when the number of
questions asked to each student is at least the cubed-logarithm of the number
of students. In an empirical study on exam data and in simulations, our
algorithm based on the maximum-likelihood estimator significantly outperforms
simple averaging in prediction accuracy and ex-post fairness even with a small
class and exam size.
Related papers
- SimGrade: Using Code Similarity Measures for More Accurate Human Grading [5.797317782326566]
We show that inaccurate and inconsistent grading of free-response programming problems is widespread in CS1 courses.
We propose several algorithms for assigning student submissions to graders, and (2) ordering submissions to maximize the probability that a grader has previously seen a similar solution.
arXiv Detail & Related papers (2024-02-19T23:06:23Z) - Computing Voting Rules with Elicited Incomplete Votes [12.981755656269097]
We study voting rules that are computable by querying voters about $t m$ candidates.
For scoring rules that are computable with limited-sized queries, we give parameterized upper and lower bounds on the number of such queries.
While there is no gap between our bounds for deterministic algorithms, identifying the exact query complexity for randomized algorithms is a challenging open problem.
arXiv Detail & Related papers (2024-02-16T22:17:01Z) - Best Arm Identification with Fixed Budget: A Large Deviation Perspective [54.305323903582845]
We present sred, a truly adaptive algorithm that can reject arms in it any round based on the observed empirical gaps between the rewards of various arms.
In particular, we present sred, a truly adaptive algorithm that can reject arms in it any round based on the observed empirical gaps between the rewards of various arms.
arXiv Detail & Related papers (2023-12-19T13:17:43Z) - Optimal Multi-Distribution Learning [88.3008613028333]
Multi-distribution learning seeks to learn a shared model that minimizes the worst-case risk across $k$ distinct data distributions.
We propose a novel algorithm that yields an varepsilon-optimal randomized hypothesis with a sample complexity on the order of (d+k)/varepsilon2.
arXiv Detail & Related papers (2023-12-08T16:06:29Z) - Regularization-Based Methods for Ordinal Quantification [49.606912965922504]
We study the ordinal case, i.e., the case in which a total order is defined on the set of n>2 classes.
We propose a novel class of regularized OQ algorithms, which outperforms existing algorithms in our experiments.
arXiv Detail & Related papers (2023-10-13T16:04:06Z) - Non-Stochastic CDF Estimation Using Threshold Queries [3.6576781735746513]
We tackle the problem of estimating an empirical distribution in a setting with two challenges.
First, the algorithm does not directly observe the data; instead, it only asks a limited number of threshold queries about each sample.
Second, the data are not assumed to be independent and identically distributed; instead, we allow for an arbitrary process generating the samples.
arXiv Detail & Related papers (2023-01-13T18:00:57Z) - Diagnostic Assessment Generation via Combinatorial Search [0.0]
We present a generic formulation of question assembly and a genetic based method that can generate assessment tests from raw problem-solving history.
Experimental results show that the proposed method outperforms greedy and random baseline by a large margin.
We also performed qualitative analysis on the generated assessment test for 9th graders.
arXiv Detail & Related papers (2021-12-06T06:19:15Z) - ProbNum: Probabilistic Numerics in Python [62.52335490524408]
Probabilistic numerical methods (PNMs) solve numerical problems via probabilistic inference.
We present ProbNum: a Python library providing state-of-the-art PNMs.
arXiv Detail & Related papers (2021-12-03T07:20:50Z) - Local policy search with Bayesian optimization [73.0364959221845]
Reinforcement learning aims to find an optimal policy by interaction with an environment.
Policy gradients for local search are often obtained from random perturbations.
We develop an algorithm utilizing a probabilistic model of the objective function and its gradient.
arXiv Detail & Related papers (2021-06-22T16:07:02Z) - Stochastic Saddle-Point Optimization for Wasserstein Barycenters [69.68068088508505]
We consider the populationimation barycenter problem for random probability measures supported on a finite set of points and generated by an online stream of data.
We employ the structure of the problem and obtain a convex-concave saddle-point reformulation of this problem.
In the setting when the distribution of random probability measures is discrete, we propose an optimization algorithm and estimate its complexity.
arXiv Detail & Related papers (2020-06-11T19:40:38Z) - Discriminative Learning via Adaptive Questioning [6.378513792050356]
We consider the problem of designing an adaptive sequence of questions that optimally classify a candidate's ability into one of several categories or discriminative grades.
A candidate's ability is modeled as an unknown parameter, which, together with the difficulty of the question asked, determines the likelihood with which s/he is able to answer a question correctly.
arXiv Detail & Related papers (2020-04-11T16:50:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.