LLMs & Legal Aid: Understanding Legal Needs Exhibited Through User Queries

• URL: http://arxiv.org/abs/2501.01711v1
• Date: Fri, 03 Jan 2025 09:12:35 GMT
• Title: LLMs & Legal Aid: Understanding Legal Needs Exhibited Through User Queries
• Abstract summary: Using GPT-4o for zero-shot classification, we categorized queries on whether users provided factual information about their issue.<n>We provide both quantitative and qualitative insight into user needs and contribute to a better understanding of user engagement with LLMs.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The paper presents a preliminary analysis of an experiment conducted by Frank Bold, a Czech expert group, to explore user interactions with GPT-4 for addressing legal queries. Between May 3, 2023, and July 25, 2023, 1,252 users submitted 3,847 queries. Unlike studies that primarily focus on the accuracy, factuality, or hallucination tendencies of large language models (LLMs), our analysis focuses on the user query dimension of the interaction. Using GPT-4o for zero-shot classification, we categorized queries on (1) whether users provided factual information about their issue (29.95%) or not (70.05%), (2) whether they sought legal information (64.93%) or advice on the course of action (35.07\%), and (3) whether they imposed requirements to shape or control the model's answer (28.57%) or not (71.43%). We provide both quantitative and qualitative insight into user needs and contribute to a better understanding of user engagement with LLMs.

Related papers

• How AI Models Manage Epistemic Authority: A Taxonomy and Comparative Analysis of Responses to User Disagreement [0.573978744524696]
  We study how a model manages its authority, referring here to its claim to knowledge, competence, or the right to advise, once a user challenges its answer.<n>We construct a new dataset of 2,310 controlled challenge scenarios and 32,340 corresponding responses from 14 models.<n>We find that models show conflicting behaviour: they validate users in 85% of responses but maintain their original claim in 65%.
  arXiv  Detail & Related papers  (2026-09-07T15:47:29Z)
• Evaluating the Hidden Costs of Personalization in Large Language Models [55.07935163900691]
  We identify three emerging risks: (1) irrelevant personalization, where models reference personal information in unnecessary contexts; (2) preference narrowing, where models reinforce informational echo chambers; and (3) sycophantic bias, where models agree excessively with user opinions.<n>We propose PRISK, a dynamic evaluation framework with automated data generation and tailored metrics that uncovers systematic limitations in current AI assistants.
  arXiv  Detail & Related papers  (2026-08-28T20:05:12Z)
• InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries [18.0624065854059]
  We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency.<n>We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes.<n>No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.
  arXiv  Detail & Related papers  (2026-08-20T16:14:47Z)
• SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information [61.41881137371855]
  We introduce SPIEval, a benchmark for large language models (LLMs)<n> SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools.<n>The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%.
  arXiv  Detail & Related papers  (2026-08-11T09:14:53Z)
• Cross-Architecture LLM Ensembles, Feature-Based Reranking and Retrieval-Augmented Prompting for Legal Information Processing [7.028604573959654]
  We report Team DU's participation in all five tasks of COLIEE 2026.<n>We use open-weight systems for legal case retrieval, case entailment, statute retrieval and entailment, and legal judgment prediction.
  arXiv  Detail & Related papers  (2026-07-13T11:07:57Z)
• ExPerT: Personalizing LLM Responses to Users' Domain Expertise via Query-Wise Semantic and Keystroke Behavioral Cues [6.879311587283136]
  We present ExPerT, a query-wise personalization framework that adapts large language models responses to users' query domain expertise.<n>ExPerT consists of two key components: (i) a semantic-behavioral expertise inference module that jointly interprets query text and keystroke dynamics.<n>Our user study with 40 participants and 1270 queries demonstrated that ExPerT reduced expertise inference error by 65.7% compared to the strongest baseline.
  arXiv  Detail & Related papers  (2026-05-04T13:21:28Z)
• SPECA: Specification-to-Checklist Agentic Auditing for Multi-Implementation Systems -- A Case Study on Ethereum Clients [1.711666249985278]
  SPECA is a Specification-to-Checklist framework that turns normative requirements into checklists.<n>We instantiate SPECA in an in-the-wild security audit contest for the Fusaka upgrade, covering 11 production clients.<n>Our improved agent, evaluated against the ground truth of a competitive audit, achieved a strict recall of 27.3 percent on high-impact vulnerabilities.
  arXiv  Detail & Related papers  (2026-02-07T12:19:00Z)
• What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models [10.883552856100684]
  We introduce HAERAE-Vision, a benchmark of 653 real-world visual questions from Korean online communities.<n>We find that even state-of-the-art models (GPT-5, Gemini 2.5 Pro) achieve under 50% on the original queries.
  arXiv  Detail & Related papers  (2026-01-07T02:33:03Z)
• Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like People [81.63702981397408]
  Given limited resources, to what extent do agents based on language models (LMs) act rationally?<n>We develop methods to benchmark and enhance agentic information-seeking, drawing on insights from human behavior.<n>For Spotter agents, our approach boosts accuracy by up to 14.7% absolute over LM-only baselines; for Captain agents, it raises expected information gain (EIG) by up to 0.227 bits (94.2% of the achievable noise ceiling)
  arXiv  Detail & Related papers  (2025-10-23T17:57:28Z)
• Teaching Language Models To Gather Information Proactively [53.85419549904644]
  Large language models (LLMs) are increasingly expected to function as collaborative partners.<n>In this work, we introduce a new task paradigm: proactive information gathering.<n>We design a scalable framework that generates partially specified, real-world tasks, masking key information.<n>Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information.
  arXiv  Detail & Related papers  (2025-07-28T23:50:09Z)
• On The Impact of Merge Request Deviations on Code Review Practices [2.8402080392117757]
  We term these cases deviations and hypothesize that ignoring them biases analytics and undermines ML models for review analysis.<n>We identify seven deviation categories, occurring in 37.02% of MRs, and propose a few-shot learning detection method (91% accuracy)<n>Our contributions include: (1) a taxonomy of MR deviations, (2) an AI-driven detection approach, and (3) empirical evidence of their impact on ML-based review analytics.
  arXiv  Detail & Related papers  (2025-06-10T14:51:20Z)
• Facts are Harder Than Opinions -- A Multilingual, Comparative Analysis of LLM-Based Fact-Checking Reliability [1.1135113962297134]
  This paper introduces a novel, dynamically data set that includes 61,514 claims in multiple languages and topics, extending existing datasets up to 2024.<n>We evaluate five prominent Large Language Models (LLMs), including GPT-4o, GPT-3.5 Turbo, LLaMA 3.1, and Mixtral 8x7B.<n>Across all models, factual-sounding claims are misclassified more often than opinions, revealing a key vulnerability.
  arXiv  Detail & Related papers  (2025-06-04T07:47:21Z)
• LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation [68.26911563519162]
  Legal consultation is essential for safeguarding individual rights and ensuring access to justice.<n>Current systems fall short in handling the interactive and knowledge-intensive nature of real-world consultations.<n>LeCoDe is a real-world multi-turn benchmark dataset comprising 3,696 legal consultation dialogues with 110,008 dialogue turns.
  arXiv  Detail & Related papers  (2025-05-26T08:24:32Z)
• LEXam: Benchmarking Legal Reasoning on 340 Law Exams [61.344330783528015]
  LEXam is a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels.<n>The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions.
  arXiv  Detail & Related papers  (2025-05-19T08:48:12Z)
• Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges [53.12387628636912]
  A crucial factor in RAG evaluation is "support", whether the information in the cited documents supports the answer. We conducted a large-scale comparative study of 45 participant submissions on 36 topics to the TREC 2024 RAG Track. Our results indicate that for 56% of the manual from-scratch assessments, human and GPT-4o predictions match perfectly.
  arXiv  Detail & Related papers  (2025-04-21T16:20:43Z)
• Retrieval-Augmented Generation with Conflicting Evidence [57.66282463340297]
  Large language model (LLM) agents are increasingly employing retrieval-augmented generation (RAG) to improve the factuality of their responses. In practice, these systems often need to handle ambiguous user queries and potentially conflicting information from multiple sources. We propose RAMDocs (Retrieval with Ambiguity and Misinformation in Documents), a new dataset that simulates complex and realistic scenarios for conflicting evidence for a user query.
  arXiv  Detail & Related papers  (2025-04-17T16:46:11Z)
• Classification or Prompting: A Case Study on Legal Requirements Traceability [4.629156733452248]
  Legal requirements traceability is a key task where engineers must analyze technical requirements against target artifacts.<n>In this paper, we investigate two automated solutions based on language models, including large ones (LLMs)<n>The first solution, Kashif, is a classifier that leverages sentence transformers and semantic similarity.<n>The second solution, RICE_LRT, prompts a recent generative LLM based on RICE, a prompt engineering framework.
  arXiv  Detail & Related papers  (2025-02-07T13:33:40Z)
• Fool Me, Fool Me: User Attitudes Toward LLM Falsehoods [13.62116438805314]
  This study examines user preferences regarding falsehood responses from Large Language Models (LLMs)<n>Surprisingly, 61% of users prefer unmarked falsehood responses over marked ones.<n>These findings suggest that user preferences, which influence LLM training via feedback mechanisms, may inadvertently encourage the generation of falsehoods.
  arXiv  Detail & Related papers  (2024-12-16T10:10:27Z)
• Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment [59.09144776166979]
  Large language models (LLMs) represent a promising, but controversial, tool in aiding scientific peer review. This study evaluates the usefulness of LLMs in a conference setting as a tool for vetting paper submissions against submission standards.
  arXiv  Detail & Related papers  (2024-11-05T18:58:00Z)
• Evaluating the Impact of a Specialized LLM on Physician Experience in Clinical Decision Support: A Comparison of Ask Avo and ChatGPT-4 [0.3999851878220878]
  Large language models (LLMs) to augment clinical decision support systems is a topic with growing interest. Current shortcomings such as hallucinations and lack of clear source citations make them unreliable for use in rapidly growing clinical environment. This study evaluates Ask Avo-derived software by AvoMD that incorporates a proprietary Model Augmented Language Retrieval system.
  arXiv  Detail & Related papers  (2024-09-06T17:53:29Z)
• Leveraging Topic Specificity and Social Relationships for Expert Finding in Community Question Answering Platforms [5.723916517485655]
  We present TUEF, a Topic-oriented User-Interaction model for Expert Finding. TUEF integrates content and social data by constructing a multi-layer graph. Experiments show TUEF outperforms all competitors with a minimum performance boost of 42.42% in P@1, 32.73% in NDCG@3, 21.76% in R@5, and 29.81% in MRR.
  arXiv  Detail & Related papers  (2024-07-04T15:50:18Z)
• Employing Label Models on ChatGPT Answers Improves Legal Text Entailment Performance [5.484345596034158]
  ChatGPT is robust in many natural language processing tasks, including legal text entailment. We use label models to integrate the provisional answers by ChatGPT into consolidated labels. The experimental results demonstrate that this approach can attain an accuracy of 76.15%, marking a significant improvement of 8.26% over the prior state-of-the-art benchmark.
  arXiv  Detail & Related papers  (2024-01-31T15:04:01Z)
• The Earth is Flat? Unveiling Factual Errors in Large Language Models [89.94270049334479]
  Large Language Models (LLMs) like ChatGPT are in various applications due to their extensive knowledge from pre-training and fine-tuning. Despite this, they are prone to generating factual and commonsense errors, raising concerns in critical areas like healthcare, journalism, and education. We introduce a novel, automatic testing framework, FactChecker, aimed at uncovering factual inaccuracies in LLMs.
  arXiv  Detail & Related papers  (2024-01-01T14:02:27Z)
• The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactions [114.67699010359637]
  We analyze a large-scale collection of real user queries to GPT. We find that tasks such as design'' and planning'' are prevalent in user interactions but are largely neglected or different from traditional NLP benchmarks.
  arXiv  Detail & Related papers  (2023-10-19T02:12:17Z)
• ExpertQA: Expert-Curated Questions and Attributed Answers [51.68314045809179]
  We conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality. We collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.
  arXiv  Detail & Related papers  (2023-09-14T16:54:34Z)
• Sentiment Analysis of Users' Reviews on COVID-19 Contact Tracing Apps with a Benchmark Dataset [6.592595861973966]
  Contact tracing has been globally adopted in the fight to control the infection rate of COVID-19. Thanks to digital technologies, such as smartphones and wearable devices, contacts of COVID-19 patients can be easily traced and informed about their potential exposure to the virus. Several interesting mobile applications have been developed. However, there are ever-growing concerns over the working mechanism and performance of these applications. In this work, we propose a pipeline starting from manual annotation via a crowd-sourcing study and concluding on the development and training of AI models for automatic sentiment analysis of users' reviews.
  arXiv  Detail & Related papers  (2021-03-01T18:43:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.