Assessing LLM Response Quality in the Context of Technology-Facilitated Abuse

• URL: http://arxiv.org/abs/2602.17672v1
• Date: Sun, 11 Jan 2026 19:26:28 GMT
• Title: Assessing LLM Response Quality in the Context of Technology-Facilitated Abuse
• Abstract summary: Technology-facilitated abuse (TFA) is a pervasive form of intimate partner violence (IPV) that leverages digital tools to control, surveil, or harm survivors.<n>We present the first expert-led manual evaluation of four large language models (LLMs)<n>We assess the quality of zero-shot single-turn LLM responses generated with a survivor safety-centered prompt on criteria tailored to the TFA domain.
• Score: 9.590490541338022
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Technology-facilitated abuse (TFA) is a pervasive form of intimate partner violence (IPV) that leverages digital tools to control, surveil, or harm survivors. While tech clinics are one of the reliable sources of support for TFA survivors, they face limitations due to staffing constraints and logistical barriers. As a result, many survivors turn to online resources for assistance. With the growing accessibility and popularity of large language models (LLMs), and increasing interest from IPV organizations, survivors may begin to consult LLM-based chatbots before seeking help from tech clinics. In this work, we present the first expert-led manual evaluation of four LLMs - two widely used general-purpose non-reasoning models and two domain-specific models designed for IPV contexts - focused on their effectiveness in responding to TFA-related questions. Using real-world questions collected from literature and online forums, we assess the quality of zero-shot single-turn LLM responses generated with a survivor safety-centered prompt on criteria tailored to the TFA domain. Additionally, we conducted a user study to evaluate the perceived actionability of these responses from the perspective of individuals who have experienced TFA. Our findings, grounded in both expert assessment and user feedback, provide insights into the current capabilities and limitations of LLMs in the TFA context and may inform the design, development, and fine-tuning of future models for this domain. We conclude with concrete recommendations to improve LLM performance for survivor support.

Related papers

• From Chasing Ghosts to Missed Attacks: Perspectives and Perceptions of SOC Practitioners on LLM Integration, Risks, and Readiness [9.803413284017138]
  Security Operations Centers (SOCs) process large volumes of security events.<n>Recent advances in Large Language Models (LLMs) suggest potential benefits for security operations.<n>Despite concerns, practitioners express a strong willingness to adopt LLMs.
  arXiv  Detail & Related papers  (2026-08-01T13:47:44Z)
• Assessing Socio-Cultural Alignment and Technical Safety of Sovereign LLMs [12.162590322796435]
  Global debate over sovereign LLMs highlights the need for governments to develop their LLMs tailored to their unique socio-cultural and historical contexts.<n>We introduce an analytic framework for extracting and evaluating the socio-cultural elements of sovereign LLMs.<n>We show that while sovereign LLMs play a meaningful role in supporting low-resource languages, they do not always meet the popular claim that these models serve their target users well.
  arXiv  Detail & Related papers  (2025-10-16T11:17:44Z)
• A Trustworthy Multi-LLM Network: Challenges,Solutions, and A Use Case [59.58213261128626]
  We propose a blockchain-enabled collaborative framework that connects multiple Large Language Models (LLMs) into a Trustworthy Multi-LLM Network (MultiLLMN)<n>This architecture enables the cooperative evaluation and selection of the most reliable and high-quality responses to complex network optimization problems.
  arXiv  Detail & Related papers  (2025-05-06T05:32:46Z)
• Should you use LLMs to simulate opinions? Quality checks for early-stage deliberation [5.4260854377259315]
  We introduce a quality control assessment designed to test the viability of LLM-simulated opinions on Likert-scale tasks.<n>This assessment comprises two key tests: emphlogical consistency and emphalignment with stakeholder expectations.<n>None of the models or methods pass the full assessment, revealing several failure modes.
  arXiv  Detail & Related papers  (2025-04-11T20:16:02Z)
• Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation [55.21013307734612]
  AoPS-Instruct is a dataset of more than 600,000 high-quality QA pairs.<n>LiveAoPSBench is an evolving evaluation set with timestamps, derived from the latest forum data.<n>Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning.
  arXiv  Detail & Related papers  (2025-01-24T06:39:38Z)
• Potential and Perils of Large Language Models as Judges of Unstructured Textual Data [0.631976908971572]
  This research investigates the effectiveness of LLM-as-judge models to evaluate the thematic alignment of summaries generated by other LLMs.<n>Our findings reveal that while LLM-as-judge offer a scalable solution comparable to human raters, humans may still excel at detecting subtle, context-specific nuances.
  arXiv  Detail & Related papers  (2025-01-14T14:49:14Z)
• Evaluating the Performance of Large Language Models in Scientific Claim Detection and Classification [0.0]
  This study evaluates the efficacy of Large Language Models (LLMs) as innovative solutions for mitigating misinformation on platforms like Twitter.<n>LLMs offer a pre-trained, adaptable approach that bypasses the extensive training and overfitting issues associated with traditional machine learning models.<n>We present a comparative analysis of LLMs' performance using a specialized dataset and propose a framework for their application in public health communication.
  arXiv  Detail & Related papers  (2024-12-21T05:02:26Z)
• Persuasion with Large Language Models: a Survey [49.86930318312291]
  Large Language Models (LLMs) have created new disruptive possibilities for persuasive communication. In areas such as politics, marketing, public health, e-commerce, and charitable giving, such LLM Systems have already achieved human-level or even super-human persuasiveness. Our survey suggests that the current and future potential of LLM-based persuasion poses profound ethical and societal risks.
  arXiv  Detail & Related papers  (2024-11-11T10:05:52Z)
• An Active Inference Strategy for Prompting Reliable Responses from Large Language Models in Medical Practice [0.0]
  Large Language Models (LLMs) are non-deterministic, may provide incorrect or harmful responses, and cannot be regulated to assure quality control. Our proposed framework refines LLM responses by restricting their primary knowledge base to domain-specific datasets containing validated medical information. We conducted a validation study where expert cognitive behaviour therapy for insomnia therapists evaluated responses from the LLM in a blind format.
  arXiv  Detail & Related papers  (2024-07-23T05:00:18Z)
• A critical review of methods and challenges in large language models [6.850038413666062]
  Review provides in-depth analysis of Large Language Models (LLMs)<n>Examines the evolution from Recurrent Neural Networks (RNNs) to Transformer models.<n>Describes state-of-the-art techniques such as in-context learning and various fine-tuning approaches.
  arXiv  Detail & Related papers  (2024-04-18T08:01:20Z)
• Privacy in Large Language Models: Attacks, Defenses and Future Directions [84.73301039987128]
  We analyze the current privacy attacks targeting large language models (LLMs) and categorize them according to the adversary's assumed capabilities. We present a detailed overview of prominent defense strategies that have been developed to counter these privacy attacks.
  arXiv  Detail & Related papers  (2023-10-16T13:23:54Z)
• Are Large Language Models Really Robust to Word-Level Perturbations? [68.60618778027694]
  We propose a novel rational evaluation approach that leverages pre-trained reward models as diagnostic tools. Longer conversations manifest the comprehensive grasp of language models in terms of their proficiency in understanding questions. Our results demonstrate that LLMs frequently exhibit vulnerability to word-level perturbations that are commonplace in daily language usage.
  arXiv  Detail & Related papers  (2023-09-20T09:23:46Z)
• Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies [104.32199881187607]
  Large language models (LLMs) have demonstrated remarkable performance across a wide array of NLP tasks. A promising approach to rectify these flaws is self-correction, where the LLM itself is prompted or guided to fix problems in its own output. This paper presents a comprehensive review of this emerging class of techniques.
  arXiv  Detail & Related papers  (2023-08-06T18:38:52Z)
• Aligning Large Language Models with Human: A Survey [53.6014921995006]
  Large Language Models (LLMs) trained on extensive textual corpora have emerged as leading solutions for a broad array of Natural Language Processing (NLP) tasks. Despite their notable performance, these models are prone to certain limitations such as misunderstanding human instructions, generating potentially biased content, or factually incorrect information. This survey presents a comprehensive overview of these alignment technologies, including the following aspects.
  arXiv  Detail & Related papers  (2023-07-24T17:44:58Z)
• A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation [21.242078120036176]
  Large Language Models (LLMs) have exploded a new heatwave of AI for their ability to engage end-users in human-level conversations. This survey concerns their safety and trustworthiness in industrial applications.
  arXiv  Detail & Related papers  (2023-05-19T02:41:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.