Entailment-Driven Privacy Policy Classification with LLMs

• URL: http://arxiv.org/abs/2409.16621v1
• Date: Wed, 25 Sep 2024 05:07:05 GMT
• Title: Entailment-Driven Privacy Policy Classification with LLMs
• Authors: Bhanuka Silva, Dishanika Denipitiyage, Suranga Seneviratne, Anirban Mahanti, Aruna Seneviratne,
• Abstract summary: We propose a framework to classify paragraphs of privacy policies into meaningful labels that are easily understood by users. Our framework improves the F1 score in average by 11.2%.
• Score: 3.564208334473993
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While many online services provide privacy policies for end users to read and understand what personal data are being collected, these documents are often lengthy and complicated. As a result, the vast majority of users do not read them at all, leading to data collection under uninformed consent. Several attempts have been made to make privacy policies more user friendly by summarising them, providing automatic annotations or labels for key sections, or by offering chat interfaces to ask specific questions. With recent advances in Large Language Models (LLMs), there is an opportunity to develop more effective tools to parse privacy policies and help users make informed decisions. In this paper, we propose an entailment-driven LLM based framework to classify paragraphs of privacy policies into meaningful labels that are easily understood by users. The results demonstrate that our framework outperforms traditional LLM methods, improving the F1 score in average by 11.2%. Additionally, our framework provides inherently explainable and meaningful predictions.

Related papers

• Privacy Policy Analysis through Prompt Engineering for LLMs [3.059256166047627]
  PAPEL (Privacy Policy Analysis through Prompt Engineering for LLMs) is a framework harnessing the power of Large Language Models (LLMs) to automate the analysis of privacy policies. It aims to streamline the extraction, annotation, and summarization of information from these policies, enhancing their accessibility and comprehensibility without requiring additional model training. We demonstrate the effectiveness of PAPEL with two applications: (i) annotation and (ii) contradiction analysis.
  arXiv  Detail & Related papers  (2024-09-23T10:23:31Z)
• Are LLM-based methods good enough for detecting unfair terms of service? [67.49487557224415]
  Large language models (LLMs) are good at parsing long text-based documents. We build a dataset consisting of 12 questions applied individually to a set of privacy policies. Some open-source models are able to provide a higher accuracy compared to some commercial models.
  arXiv  Detail & Related papers  (2024-08-24T09:26:59Z)
• Data Exposure from LLM Apps: An In-depth Investigation of OpenAI's GPTs [17.433387980578637]
  This paper aims to bring transparency in data practices of LLM apps. We study OpenAI's GPT app ecosystem. We find that Actions collect expansive data about users, including sensitive information prohibited by OpenAI, such as passwords.
  arXiv  Detail & Related papers  (2024-08-23T17:42:06Z)
• LLM-PBE: Assessing Data Privacy in Large Language Models [111.58198436835036]
  Large Language Models (LLMs) have become integral to numerous domains, significantly advancing applications in data management, mining, and analysis. Despite the critical nature of this issue, there has been no existing literature to offer a comprehensive assessment of data privacy risks in LLMs. Our paper introduces LLM-PBE, a toolkit crafted specifically for the systematic evaluation of data privacy risks in LLMs.
  arXiv  Detail & Related papers  (2024-08-23T01:37:29Z)
• Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning [62.224804688233]
  differential privacy (DP) offers a promising solution by ensuring models are 'almost indistinguishable' with or without any particular privacy unit. We study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users.
  arXiv  Detail & Related papers  (2024-06-20T13:54:32Z)
• Automated Detection and Analysis of Data Practices Using A Real-World Corpus [20.4572759138767]
  We propose an automated approach to identify and visualize data practices within privacy policies at different levels of detail. Our approach accurately matches data practice descriptions with policy excerpts, facilitating the presentation of simplified privacy information to users.
  arXiv  Detail & Related papers  (2024-02-16T18:51:40Z)
• PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners [81.571305826793]
  We introduce Contextual Privacy Protection Language Models (PrivacyMind) Our work offers a theoretical analysis for model design and benchmarks various techniques. In particular, instruction tuning with both positive and negative examples stands out as a promising method.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• PolicyGPT: Automated Analysis of Privacy Policies with Large Language Models [41.969546784168905]
  In practical use, users tend to click the Agree button directly rather than reading them carefully. This practice exposes users to risks of privacy leakage and legal issues. Recently, the advent of Large Language Models (LLM) such as ChatGPT and GPT-4 has opened new possibilities for text analysis.
  arXiv  Detail & Related papers  (2023-09-19T01:22:42Z)
• PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English [77.79102359580702]
  We introduce the Privacy Policy Language Understanding Evaluation benchmark, a multi-task benchmark for evaluating the privacy policy language understanding. We also collect a large corpus of privacy policies to enable privacy policy domain-specific language model pre-training. We demonstrate that domain-specific continual pre-training offers performance improvements across all tasks.
  arXiv  Detail & Related papers  (2022-12-20T05:58:32Z)
• Retrieval Enhanced Data Augmentation for Question Answering on Privacy Policies [74.01792675564218]
  We develop a data augmentation framework based on ensembling retriever models that captures relevant text segments from unlabeled policy documents. To improve the diversity and quality of the augmented data, we leverage multiple pre-trained language models (LMs) and cascade them with noise reduction filter models. Using our augmented data on the PrivacyQA benchmark, we elevate the existing baseline by a large margin (10% F1) and achieve a new state-of-the-art F1 score of 50%.
  arXiv  Detail & Related papers  (2022-04-19T15:45:23Z)
• A Comparative Study of Sequence Classification Models for Privacy Policy Coverage Analysis [0.0]
  Privacy policies are legal documents that describe how a website will collect, use, and distribute a user's data. Our solution is to provide users with a coverage analysis of a given website's privacy policy using a wide range of classical machine learning and deep learning techniques.
  arXiv  Detail & Related papers  (2020-02-12T21:46:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.