All Eyes on the Ranker: Participatory Auditing to Surface Blind Spots in Ranked Search Results

• URL: http://arxiv.org/abs/2604.09946v1
• Date: Fri, 10 Apr 2026 22:55:03 GMT
• Title: All Eyes on the Ranker: Participatory Auditing to Surface Blind Spots in Ranked Search Results
• Abstract summary: This paper argues that participatory auditing is essential for revealing users' causal and contextual understandings of how ranked search results produce impacts.<n>We report on three participatory auditing workshops, exploring varying levels of transparency and user controls, and examining an intentionally adversarially manipulated ranking.
• Score: 20.872574833329217
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Search engines that present users with a ranked list of search results are a fundamental technology for providing public access to information. Evaluations of such systems are typically conducted by domain experts and focus on model-centric metrics, relevance judgments, or output-based analyses, rather than on how accountability, harm, or trust are experienced by users. This paper argues that participatory auditing is essential for revealing users' causal and contextual understandings of how ranked search results produce impacts, particularly as ranking models appear increasingly convincing and sophisticated in their semantic interpretation of user queries. We report on three participatory auditing workshops (n=21) in which participants engaged with a custom search interface across four tasks, comparing a lexical ranker (BM25) and a neural semantic reranker (MonoT5), exploring varying levels of transparency and user controls, and examining an intentionally adversarially manipulated ranking. Reflexive activities prompted participants to articulate causal narratives linking search system properties to broader impacts. Synthesising the findings, we contribute a taxonomy of user-perceived impacts of ranked search results, spanning epistemic, representational, infrastructural, and downstream social impacts. However, interactions with the neural model revealed limits to participatory auditing itself: perceived system competence and accumulated trust reduced critical scrutiny during the workshop, allowing manipulations to go undetected. Participants expressed desire for visibility into the full search pipeline and recourse mechanisms. Together, these findings show how participatory auditing can surface user perceived impacts and accountability gaps that remain unseen when relying on conventional audits, while revealing where participatory auditing may encounter limitations.

Related papers

• Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters [3.8028536988925477]
  We present three experiments that study the effects of interactive feedback collections on user impressions in domains with objective and subjective feedback.<n>Results show that in a context where there is an objectively correct answer, providing HITL feedback lowered both participants' trust in the system and their perception of system accuracy.<n>These results highlight the importance of considering the effects of allowing different types of end-user feedback on user trust when designing intelligent systems.
  arXiv  Detail & Related papers  (2026-07-20T04:46:25Z)
• Seek and You Shall Find: Design & Evaluation of a Context-Aware Interactive Search Companion [2.901725877154321]
  We present an interactive search companion that seamlessly integrates expert search strategies into existing search engine result pages.<n>Providing context-aware tips on clarifying information needs, improving query formulation, and encouraging result exploration, our companion aims to foster reflective search behaviour.<n>A user study demonstrates the companion's successful encouragement of more active and exploratory search, leading users to submit 75 % more queries and view roughly twice as many results.
  arXiv  Detail & Related papers  (2026-01-16T13:39:33Z)
• Addressing Personalized Bias for Unbiased Learning to Rank [56.663619153713434]
  Unbiased learning to rank (ULTR) aims to learn unbiased ranking models from biased user behavior logs.<n>We propose a novel user-aware inverse-propensity-score estimator for learning-to-rank objectives.
  arXiv  Detail & Related papers  (2025-08-28T14:01:31Z)
• Understanding Visual Saliency of Outlier Items in Product Search [62.12411635661447]
  In two-sided marketplaces, items compete for user attention, which translates to revenue for suppliers.<n>Recent work suggests that inter-item dependencies, such as outlier items in a ranking, also affect item exposure.<n>We investigate how top-down factors influence users' perception of item outlierness in a realistic online shopping scenario.
  arXiv  Detail & Related papers  (2025-03-30T21:22:23Z)
• Interactive Counterfactual Exploration of Algorithmic Harms in Recommender Systems [3.990406494980651]
  This study introduces an interactive tool designed to help users comprehend and explore the impacts of algorithmic harms in recommender systems. By leveraging visualizations, counterfactual explanations, and interactive modules, the tool allows users to investigate how biases such as miscalibration affect their recommendations.
  arXiv  Detail & Related papers  (2024-09-10T23:58:27Z)
• Towards Detecting and Mitigating Cognitive Bias in Spoken Conversational Search [14.916529791823868]
  This paper draws upon insights from information seeking, psychology, cognitive science, and wearable sensors to provoke novel conversations in the community. We propose a framework including multimodal instruments and methods for experimental designs and settings.
  arXiv  Detail & Related papers  (2024-05-21T03:50:32Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• Decoy Effect In Search Interaction: Understanding User Behavior and Measuring System Vulnerability [33.78769577114657]
  The study explores how decoy results alter users' interactions on search engine result pages. It introduces the DEJA-VU metric to assess systems' susceptibility to the decoy effect. The results show differences in systems' effectiveness and vulnerability.
  arXiv  Detail & Related papers  (2024-03-27T11:20:48Z)
• Navigating the Thin Line: Examining User Behavior in Search to Detect Engagement and Backfire Effects [0.0]
  We investigate whether different levels of bias metrics and search results presentation can affect the stance diversity consumption and search behavior of opinionated users. Our results show that exposing participants to (counter-attitudinally) biased search results increases their consumption of attitude-opposing content. We also found that bias was associated with a trend toward overall fewer interactions within the search page.
  arXiv  Detail & Related papers  (2024-01-20T10:28:25Z)
• Measuring the Effect of Influential Messages on Varying Personas [67.1149173905004]
  We present a new task, Response Forecasting on Personas for News Media, to estimate the response a persona might have upon seeing a news message. The proposed task not only introduces personalization in the modeling but also predicts the sentiment polarity and intensity of each response. This enables more accurate and comprehensive inference on the mental state of the persona.
  arXiv  Detail & Related papers  (2023-05-25T21:01:00Z)
• Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey [71.43956423427397]
  We aim to identify the nonverbal cues and computational methodologies resulting in effective performance. This survey differs from its counterparts by involving the widest spectrum of social phenomena and interaction settings. Some major observations are: the most often used nonverbal cue, computational method, interaction environment, and sensing approach are speaking activity, support vector machines, and meetings composed of 3-4 persons equipped with microphones and cameras, respectively.
  arXiv  Detail & Related papers  (2022-07-20T13:37:57Z)
• Towards Unbiased Visual Emotion Recognition via Causal Intervention [63.74095927462]
  We propose a novel Emotion Recognition Network (IERN) to alleviate the negative effects brought by the dataset bias. A series of designed tests validate the effectiveness of IERN, and experiments on three emotion benchmarks demonstrate that IERN outperforms other state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-07-26T10:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.