Fairness Audits of Institutional Risk Models in Deployed ML Pipelines

• URL: http://arxiv.org/abs/2604.19468v1
• Date: Tue, 21 Apr 2026 13:50:43 GMT
• Title: Fairness Audits of Institutional Risk Models in Deployed ML Pipelines
• Abstract summary: We present a replica-based audit of a deployed Early Warning System (EWS) using institutional training data and design specifications.<n>We evaluate disparities by gender, age, and residency status across the full pipeline.<n>Post-processing amplifies these disparities by collapsing heterogeneous probabilities into percentile-based risk tiers.
• Score: 17.015521443271115
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Fairness audits of institutional risk models are critical for understanding how deployed machine learning pipelines allocate resources. Drawing on multi-year collaboration with Centennial College, where our prior ethnographic work introduced the ASP-HEI Cycle, we present a replica-based audit of a deployed Early Warning System (EWS), replicating its model using institutional training data and design specifications. We evaluate disparities by gender, age, and residency status across the full pipeline (training data, model predictions, and post-processing) using standard fairness metrics. Our audit reveals systematic misallocation: younger, male, and international students are disproportionately flagged for support, even when many ultimately succeed, while older and female students with comparable dropout risk are under-identified. Post-processing amplifies these disparities by collapsing heterogeneous probabilities into percentile-based risk tiers. This work provides a replicable methodology for auditing institutional ML systems and shows how disparities emerge and compound across stages, highlighting the importance of evaluating construct validity alongside statistical fairness. It contributes one empirical thread to a broader program investigating algorithms, student data, and power in higher education.

Related papers

• A Full-Pipeline Framework for Evaluating Membership Inference Attacks in Machine Learning [31.096361601691708]
  We introduce a comprehensive evaluation framework that characterizes privacy risks across the entire machine learning pipeline.<n>Our framework rigorously evaluates state-of-the-art MIAs across a broad spectrum of training configurations.
  arXiv  Detail & Related papers  (2026-05-28T06:48:22Z)
• Reliable and Reproducible Demographic Inference for Fairness in Face Analysis [63.46525489354455]
  We propose a fully reproducible DAI pipeline that replaces conventional end-to-end training with a modular transfer learning approach.<n>We audit this pipeline across three dimensions: accuracy, fairness, and a newly introduced notion of robustness, defined via intra-identity consistency.<n>Our results show that the proposed method outperforms strong baselines, particularly on ethnicity, which is the more challenging attribute.
  arXiv  Detail & Related papers  (2025-10-23T12:22:02Z)
• Evaluating Federated Learning for At-Risk Student Prediction: A Comparative Analysis of Model Complexity and Data Balancing [0.07161783472741748]
  This study proposes and validates a Federated Learning (FL) framework to proactively identify at-risk students.<n>FL is a practical and scalable solution for early-warning systems that inherently respects student data sovereignty.
  arXiv  Detail & Related papers  (2025-08-23T19:58:16Z)
• Addressing Bias in LLMs: Strategies and Application to Fair AI-based Recruitment [49.81946749379338]
  This work seeks to analyze the capacity of Transformers-based systems to learn demographic biases present in the data.<n>We propose a privacy-enhancing framework to reduce gender information from the learning pipeline as a way to mitigate biased behaviors in the final tools.
  arXiv  Detail & Related papers  (2025-06-13T15:29:43Z)
• Modelling higher education dropouts using sparse and interpretable post-clustering logistic regression [0.8437187555622164]
  Higher education dropout constitutes a critical challenge for tertiary education systems worldwide.<n>The model introduced in this paper is a specialized form of logistic regression, specifically adapted to the context of university dropout analysis.
  arXiv  Detail & Related papers  (2025-05-12T14:05:23Z)
• Meta-Statistical Learning: Supervised Learning of Statistical Inference [59.463430294611626]
  This work demonstrates that the tools and principles driving the success of large language models (LLMs) can be repurposed to tackle distribution-level tasks.<n>We propose meta-statistical learning, a framework inspired by multi-instance learning that reformulates statistical inference tasks as supervised learning problems.
  arXiv  Detail & Related papers  (2025-02-17T18:04:39Z)
• Data vs. Model Machine Learning Fairness Testing: An Empirical Study [23.535630175567146]
  We take the first steps towards evaluating a more holistic approach by testing for fairness both before and after model training. We evaluate the effectiveness of the proposed approach using an empirical analysis of the relationship between model dependent and independent fairness metrics. Our results indicate that testing for fairness prior to training can be a cheap'' and effective means of catching a biased data collection process early.
  arXiv  Detail & Related papers  (2024-01-15T14:14:16Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Cross-Institutional Transfer Learning for Educational Models: Implications for Model Performance, Fairness, and Equity [8.762925492954777]
  This study investigates cross-institutional learning via an empirical case study in higher education. We examine the feasibility of cross-institutional transfer under real-world data- and model-sharing constraints. We find no evidence of a fairness-accuracy tradeoff across dozens of models and transfer schemes evaluated.
  arXiv  Detail & Related papers  (2023-05-01T16:27:49Z)
• A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems [128.63953314853327]
  "Lifelong Learning" systems are capable of 1) Continuous Learning, 2) Transfer and Adaptation, and 3) Scalability. We show that this suite of metrics can inform the development of varied and complex Lifelong Learning systems.
  arXiv  Detail & Related papers  (2023-01-18T21:58:54Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Detecting discriminatory risk through data annotation based on Bayesian inferences [5.017973966200985]
  We propose a method of data annotation that aims to warn about the risk of discriminatory results of a given data set. We empirically test our system on three datasets commonly accessed by the machine learning community.
  arXiv  Detail & Related papers  (2021-01-27T12:43:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.