Utilizing LLMs to Investigate the Disputed Role of Evidence in Electronic Cigarette Health Policy Formation in Australia and the UK

• URL: http://arxiv.org/abs/2505.06782v1
• Date: Sat, 10 May 2025 23:40:28 GMT
• Title: Utilizing LLMs to Investigate the Disputed Role of Evidence in Electronic Cigarette Health Policy Formation in Australia and the UK
• Abstract summary: Australia and the UK have developed contrasting approaches to the regulation of electronic cigarettes.<n>We developed and evaluated a Large Language Model-based sentence classifier to perform automated analyses of electronic cigarette-related policy documents.
• Score: 0.7646713951724013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Australia and the UK have developed contrasting approaches to the regulation of electronic cigarettes, with - broadly speaking - Australia adopting a relatively restrictive approach and the UK adopting a more permissive approach. Notably, these divergent policies were developed from the same broad evidence base. In this paper, to investigate differences in how the two jurisdictions manage and present evidence, we developed and evaluated a Large Language Model-based sentence classifier to perform automated analyses of electronic cigarette-related policy documents drawn from official Australian and UK legislative processes (109 documents in total). Specifically, we utilized GPT-4 to automatically classify sentences based on whether they contained claims that e-cigarettes were broadly helpful or harmful for public health. Our LLM-based classifier achieved an F-score of 0.9. Further, when applying the classifier to our entire sentence-level corpus, we found that Australian legislative documents show a much higher proportion of harmful statements, and a lower proportion of helpful statements compared to the expected values, with the opposite holding for the UK. In conclusion, this work utilized an LLM-based approach to provide evidence to support the contention that - drawing on the same evidence base - Australian ENDS-related policy documents emphasize the harms associated with ENDS products and UK policy documents emphasize the benefits. Further, our approach provides a starting point for using LLM-based methods to investigate the complex relationship between evidence and health policy formation.

Related papers

• Evaluating RAG for French immigration law: a benchmark and baseline study [0.3058685580689604]
  International recruitment in France requires navigating a layered legal framework absent from existing legal AI benchmarks.<n>We present a publicly available benchmark and first comparative evaluation for this domain, covering permit-type recommendation, required-document retrieval, and legal citation coverage.
  arXiv  Detail & Related papers  (2026-07-27T13:55:35Z)
• Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering [54.97384993676565]
  We formalize Regulatory Compliance QA with RegOps-Bench, a novel benchmark featuring an Operational Knowledge Graph derived from complex national R&D regulations.<n>RefWalk traverses cross-document citations, fuses multi-view candidates via max-based aggregation, and enforces per-rule attribution to explicitly map claims to sources.
  arXiv  Detail & Related papers  (2026-05-28T10:38:38Z)
• Multi-Sourced, Multi-Agent Evidence Retrieval for Fact-Checking [47.47518672198846]
  Misinformation spreading over the Internet poses a significant threat to both societies and individuals.<n>Previous methods rely on semantic and social-contextual patterns learned from training data.<n>We propose WKGFC, which exploits authorized open knowledge graph as a core resource of evidence.
  arXiv  Detail & Related papers  (2026-02-27T19:29:01Z)
• LegalOne: A Family of Foundation Models for Reliable Legal Reasoning [54.57434222018289]
  We present LegalOne, a family of foundational models specifically tailored for the Chinese legal domain.<n>LegalOne is developed through a comprehensive three-phase pipeline designed to master legal reasoning.<n>We publicly release the LegalOne weights and the LegalKit evaluation framework to advance the field of Legal AI.
  arXiv  Detail & Related papers  (2026-01-31T10:18:32Z)
• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
  We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-23T11:36:10Z)
• Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts [54.15982476754607]
  Large language models (LLMs) are now deployed at unprecedented scale, assisting millions of users in daily tasks.<n>This study defines complicit facilitation as the provision of guidance or support that enables illicit user instructions.<n>Using real-world legal cases and established legal frameworks, we construct an evaluation benchmark spanning 269 illicit scenarios and 50 illicit intents.
  arXiv  Detail & Related papers  (2025-11-25T16:01:31Z)
• Universal Legal Article Prediction via Tight Collaboration between Supervised Classification Model and LLM [42.11889345473452]
  Legal Article Prediction (LAP) is a critical task in legal text classification.<n>We propose Uni-LAP, a universal framework for legal article prediction.
  arXiv  Detail & Related papers  (2025-09-26T09:42:20Z)
• Augmented Question-guided Retrieval (AQgR) of Indian Case Law with LLM, RAG, and Structured Summaries [0.0]
  This paper proposes the use of Large Language Models (LLMs) to facilitate the retrieval of relevant cases.<n>Our approach combines Retrieval Augmented Generation (RAG) with structured summaries optimized for Indian case law.<n>The system generates targeted legal questions based on factual scenarios to identify relevant case law more effectively.
  arXiv  Detail & Related papers  (2025-07-23T05:24:44Z)
• Computational Identification of Regulatory Statements in EU Legislation [0.0]
  A computational method is valuable for scaling the identification of such statements from a growing body of EU legislation.<n>We provide a specific definition for what constitutes a regulatory statement based on the institutional grammar tool.
  arXiv  Detail & Related papers  (2025-05-01T12:11:32Z)
• Machine Learning-Driven Convergence Analysis in Multijurisdictional Compliance Using BERT and K-Means Clustering [0.0]
  This study compares various regulations to identify areas where they overlap or diverge.<n>International companies can learn valuable lessons from this report.<n>The study's objective is to "bridge the gap between legal knowledge and technical expertise"
  arXiv  Detail & Related papers  (2025-01-23T22:11:18Z)
• Knowledge-Augmented Reasoning for EUAIA Compliance and Adversarial Robustness of LLMs [1.368472250332885]
  The EU AI Act (EUAIA) introduces requirements for AI systems which intersect with the processes required to establish adversarial robustness. This paper presents a functional architecture that focuses on bridging the two properties. We aim to support developers and auditors with a reasoning layer based on knowledge augmentation.
  arXiv  Detail & Related papers  (2024-10-04T18:23:14Z)
• InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
  InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws. We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries. InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
  arXiv  Detail & Related papers  (2024-06-21T06:19:03Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)
• LM-Polygraph: Uncertainty Estimation for Language Models [71.21409522341482]
  Uncertainty estimation (UE) methods are one path to safer, more responsible, and more effective use of large language models (LLMs) We introduce LM-Polygraph, a framework with implementations of a battery of state-of-the-art UE methods for LLMs in text generation tasks, with unified program interfaces in Python. It introduces an extendable benchmark for consistent evaluation of UE techniques by researchers, and a demo web application that enriches the standard chat dialog with confidence scores.
  arXiv  Detail & Related papers  (2023-11-13T15:08:59Z)
• A Hierarchical Neural Framework for Classification and its Explanation in Large Unstructured Legal Documents [0.5812284760539713]
  We define this problem as "scarce annotated legal documents" We propose a deep-learning-based classification framework which we call MESc. We also propose an explanation extraction algorithm named ORSE.
  arXiv  Detail & Related papers  (2023-09-19T12:18:28Z)
• Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs [65.9077733300329]
  Large Language Models (LLMs) have the potential to greatly enhance the analysis of public affairs documents. LLMs can be of great use to process domain-specific documents, such as those in the domain of public affairs.
  arXiv  Detail & Related papers  (2023-06-05T13:35:01Z)
• Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset [46.156169284961045]
  We offer an approach to filtering grounded in law, which has directly addressed the tradeoffs in filtering material. First, we gather and make available the Pile of Law, a 256GB dataset of open-source English-language legal and administrative data. Second, we distill the legal norms that governments have developed to constrain the inclusion of toxic or private content into actionable lessons. Third, we show how the Pile of Law offers researchers the opportunity to learn such filtering rules directly from the data.
  arXiv  Detail & Related papers  (2022-07-01T06:25:15Z)
• Eider: Evidence-enhanced Document-level Relation Extraction [56.71004595444816]
  Document-level relation extraction (DocRE) aims at extracting semantic relations among entity pairs in a document. We propose a three-stage evidence-enhanced DocRE framework consisting of joint relation and evidence extraction, evidence-centered relation extraction (RE), and fusion of extraction results.
  arXiv  Detail & Related papers  (2021-06-16T09:43:16Z)
• Towards Automatic Comparison of Data Privacy Documents: A Preliminary Experiment on GDPR-like Laws [1.3537117504260623]
  General Data Protection Regulation (NLP) becomes standard law for protection in many countries. 12 countries adopt their similarities-like regulations, but evaluating differences is time-consuming and needs manual effort from legal experts. In this paper, we investigate a simple natural language processing (NLP) approach to tackle the problem.
  arXiv  Detail & Related papers  (2021-05-21T03:59:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.