Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs

• URL: http://arxiv.org/abs/2306.02864v2
• Date: Tue, 8 Aug 2023 09:48:36 GMT
• Title: Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs
• Authors: Alejandro Pe\~na, Aythami Morales, Julian Fierrez, Ignacio Serna, Javier Ortega-Garcia, I\~nigo Puente, Jorge Cordova, Gonzalo Cordova
• Abstract summary: Large Language Models (LLMs) have the potential to greatly enhance the analysis of public affairs documents. LLMs can be of great use to process domain-specific documents, such as those in the domain of public affairs.
• Score: 65.9077733300329
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The analysis of public affairs documents is crucial for citizens as it promotes transparency, accountability, and informed decision-making. It allows citizens to understand government policies, participate in public discourse, and hold representatives accountable. This is crucial, and sometimes a matter of life or death, for companies whose operation depend on certain regulations. Large Language Models (LLMs) have the potential to greatly enhance the analysis of public affairs documents by effectively processing and understanding the complex language used in such documents. In this work, we analyze the performance of LLMs in classifying public affairs documents. As a natural multi-label task, the classification of these documents presents important challenges. In this work, we use a regex-powered tool to collect a database of public affairs documents with more than 33K samples and 22.5M tokens. Our experiments assess the performance of 4 different Spanish LLMs to classify up to 30 different topics in the data in different configurations. The results shows that LLMs can be of great use to process domain-specific documents, such as those in the domain of public affairs.

Related papers

• Large Language Models for Judicial Entity Extraction: A Comparative Study [0.0]
  This research investigates the application of Large Language Models in identifying domain-specific entities within case law documents. The study evaluates the performance of state-of-the-art Large Language Model architectures, including Large Language Model Meta AI 3, Mistral, and Gemma.
  arXiv  Detail & Related papers  (2024-07-08T09:49:03Z)
• Quantifying Generative Media Bias with a Corpus of Real-world and Generated News Articles [12.356251871670011]
  Large language models (LLMs) are increasingly being utilised across a range of tasks and domains. This study focuses on political bias, detecting it using both supervised models and LLMs. For the first time within the journalistic domain, this study outlines a framework for quantifiable experiments.
  arXiv  Detail & Related papers  (2024-06-16T01:32:04Z)
• LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset [0.0]
  This study addresses the gap in the literature working with large legal corpora about how to isolate cases, in our case summary judgments, from a large corpus of UK court decisions. We use the Cambridge Law Corpus of 356,011 UK court decisions and determine that the large language model achieves a weighted F1 score of 0.94 versus 0.78 for keywords. We identify and extract 3,102 summary judgment cases, enabling us to map their distribution across various UK courts over a temporal span.
  arXiv  Detail & Related papers  (2024-03-04T10:13:30Z)
• L-Eval: Instituting Standardized Evaluation for Long Context Language Models [91.05820785008527]
  We propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) We build a new evaluation suite containing 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs. Results show that popular n-gram matching metrics generally can not correlate well with human judgment.
  arXiv  Detail & Related papers  (2023-07-20T17:59:41Z)
• One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support [18.810320088441678]
  This work introduces a novel NLP benchmark for the legal domain. It challenges LLMs in five key dimensions: processing emphlong documents (up to 50K tokens), using emphdomain-specific knowledge (embodied in legal texts) and emphmultilingual understanding (covering five languages) Our benchmark contains diverse datasets from the Swiss legal system, allowing for a comprehensive study of the underlying non-English, inherently multilingual legal system.
  arXiv  Detail & Related papers  (2023-06-15T16:19:15Z)
• Document Layout Annotation: Database and Benchmark in the Domain of Public Affairs [62.38140271294419]
  We propose a procedure to semi-automatically annotate digital documents with different layout labels. We collect a novel database for DLA in the public affairs domain using a set of 24 data sources from the Spanish Administration. The results of our experiments validate the proposed text labeling procedure with accuracy up to 99%.
  arXiv  Detail & Related papers  (2023-06-12T08:21:50Z)
• Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
  This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
  arXiv  Detail & Related papers  (2023-05-24T10:45:25Z)
• ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models [78.08792285698853]
  We present a large-scale empirical study on general language ability evaluation of pretrained language models (ElitePLM) Our empirical results demonstrate that: (1) PLMs with varying training objectives and strategies are good at different ability tests; (2) fine-tuning PLMs in downstream tasks is usually sensitive to the data size and distribution; and (3) PLMs have excellent transferability between similar tasks.
  arXiv  Detail & Related papers  (2022-05-03T14:18:10Z)
• Algorithmic Fairness Datasets: the Story so Far [68.45921483094705]
  Data-driven algorithms are studied in diverse domains to support critical decisions, directly impacting people's well-being. A growing community of researchers has been investigating the equity of existing algorithms and proposing novel ones, advancing the understanding of risks and opportunities of automated decision-making for historically disadvantaged populations. Progress in fair Machine Learning hinges on data, which can be appropriately used only if adequately documented. Unfortunately, the algorithmic fairness community suffers from a collective data documentation debt caused by a lack of information on specific resources (opacity) and scatteredness of available information (sparsity)
  arXiv  Detail & Related papers  (2022-02-03T17:25:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.