Responsible Intelligence in Practice: A Fairness Audit of Open Large Language Models for Library Reference Services

• URL: http://arxiv.org/abs/2602.18935v1
• Date: Sat, 21 Feb 2026 19:05:03 GMT
• Title: Responsible Intelligence in Practice: A Fairness Audit of Open Large Language Models for Library Reference Services
• Abstract summary: Large language models (LLMs) offer potential for broadening access to information assistance, but may also reproduce societal biases embedded in their training data.<n>We apply a systematic evaluation approach that combines diagnostic classification to detect systematic differences with linguistic analysis to interpret their sources.<n>We find no compelling evidence of systematic differentiation by race/ethnicity, and only minor evidence of sex-linked differentiation in one model.
• Score: 1.9402967965814402
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: As libraries explore large language models (LLMs) as a scalable layer for reference services, a core fairness question follows: can LLM-based services support all patrons fairly, regardless of demographic identity? While LLMs offer great potential for broadening access to information assistance, they may also reproduce societal biases embedded in their training data, potentially undermining libraries' commitments to impartial service. In this chapter, we apply a systematic evaluation approach that combines diagnostic classification to detect systematic differences with linguistic analysis to interpret their sources. Across three widely used open models (Llama-3.1 8B, Gemma-2 9B, and Ministral 8B), we find no compelling evidence of systematic differentiation by race/ethnicity, and only minor evidence of sex-linked differentiation in one model. We discuss implications for responsible AI adoption in libraries and the importance of ongoing monitoring in aligning LLM-based services with core professional values.

Related papers

• Classifying German Language Proficiency Levels Using Large Language Models [0.24683296459020942]
  This paper investigates the use of Large Language Models (LLMs) for automatically classifying German texts into different proficiency levels.<n>To support robust training and evaluation, we construct a diverse dataset by combining multiple existing CEFR-annotated corpora with synthetic data.<n>Our results show a consistent performance improvement over prior methods, highlighting the potential of LLMs for reliable and scalable CEFR classification.
  arXiv  Detail & Related papers  (2025-12-06T16:15:45Z)
• SoK: Large Language Model Copyright Auditing via Fingerprinting [69.14570598973195]
  We introduce a unified framework and formal taxonomy that categorizes existing methods into white-box and black-box approaches.<n>We propose LeaFBench, the first systematic benchmark for evaluating LLM fingerprinting under realistic deployment scenarios.
  arXiv  Detail & Related papers  (2025-08-27T12:56:57Z)
• Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length Contexts [19.640586886024952]
  HAMLET is a framework for evaluating the long-context comprehension of large language models.<n>It structures texts into a three-level key-fact hierarchy at root, branch, and leaf-levels.<n>It employs query-focused summarization to evaluate how well models recall and faithfully represent information at each level.
  arXiv  Detail & Related papers  (2025-08-27T05:23:22Z)
• Fairness Evaluation of Large Language Models in Academic Library Reference Services [6.335631290002225]
  We evaluate whether large language models (LLMs) differentiate responses across user identities by prompting six state-of-the-art LLMs to assist patrons differing in sex, race/ethnicity, and institutional role.<n>We found no evidence of differentiation by race or ethnicity, and only minor evidence of stereotypical bias against women in one model.<n>These findings suggest that current LLMs show a promising degree of readiness to support equitable and contextually appropriate communication in academic library reference services.
  arXiv  Detail & Related papers  (2025-07-06T03:28:24Z)
• How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective [64.00022624183781]
  Large language models (LLMs) can assess relevance and support information retrieval (IR) tasks.<n>We investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability.
  arXiv  Detail & Related papers  (2025-04-10T16:14:55Z)
• Analyzing Fairness of Classification Machine Learning Model with Structured Dataset [1.0923877073891446]
  This study investigates the fairness of machine learning models applied to structured datasets in classification tasks.<n>Three fairness libraries; Fairlearn by Microsoft, AIF360 by IBM, and the What If Tool by Google were employed.<n>The research aims to assess the extent of bias in the ML models, compare the effectiveness of these libraries, and derive actionable insights for practitioners.
  arXiv  Detail & Related papers  (2024-12-13T06:31:09Z)
• FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition [56.76951887823882]
  Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. We present FAC$2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation.
  arXiv  Detail & Related papers  (2024-02-29T21:05:37Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
  arXiv  Detail & Related papers  (2023-12-26T07:24:46Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.