Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence

• URL: http://arxiv.org/abs/2306.07075v1
• Date: Mon, 12 Jun 2023 12:40:48 GMT
• Title: Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence
• Authors: John J. Nay, David Karamardian, Sarah B. Lawsky, Wenting Tao, Meghana Bhat, Raghav Jain, Aaron Travis Lee, Jonathan H. Choi, Jungo Kasai
• Abstract summary: This paper explores Large Language Models' (LLMs) capabilities in applying tax law. Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release. Findings indicate that LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy but not yet at expert tax lawyer levels.
• Score: 5.07013500385659
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Better understanding of Large Language Models' (LLMs) legal analysis abilities can contribute to improving the efficiency of legal services, governing artificial intelligence, and leveraging LLMs to identify inconsistencies in law. This paper explores LLM capabilities in applying tax law. We choose this area of law because it has a structure that allows us to set up automated validation pipelines across thousands of examples, requires logical reasoning and maths skills, and enables us to test LLM capabilities in a manner relevant to real-world economic lives of citizens and companies. Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release. We experiment with retrieving and utilising the relevant legal authority to assess the impact of providing additional legal context to LLMs. Few-shot prompting, presenting examples of question-answer pairs, is also found to significantly enhance the performance of the most advanced model, GPT-4. The findings indicate that LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy but not yet at expert tax lawyer levels. As LLMs continue to advance, their ability to reason about law autonomously could have significant implications for the legal profession and AI governance.

Related papers

• InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
  InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws. We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries. InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
  arXiv  Detail & Related papers  (2024-06-21T06:19:03Z)
• Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey [1.0770079992809338]
  The capabilities of Large Language Models (LLMs) are increasingly demonstrating unique roles in the legal sector. This survey delves into the synergy between LLMs and the legal system, such as their applications in tasks like legal text comprehension, case retrieval, and analysis. The survey showcases the latest advancements in fine-tuned legal LLMs tailored for various legal systems, along with legal datasets available for fine-tuning LLMs in various languages.
  arXiv  Detail & Related papers  (2024-04-01T08:35:56Z)
• BLT: Can Large Language Models Handle Basic Legal Text? [50.46167465931653]
  GPT-4, Claude, and PaLM 2 perform poorly at basic legal text handling. Fine-tuning for these tasks brings even a smaller model to near-perfect performance on our test set.
  arXiv  Detail & Related papers  (2023-11-16T09:09:22Z)
• A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction [60.70089334782383]
  Large language models (LLMs) have demonstrated great potential for domain-specific applications. Recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks. We design practical baseline solutions based on LLMs and test on the task of legal judgment prediction.
  arXiv  Detail & Related papers  (2023-10-18T07:38:04Z)
• Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration [52.57055162778548]
  Legal Judgment Prediction (LJP) has become an increasingly crucial task in Legal AI. Precedents are the previous legal cases with similar facts, which are the basis for the judgment of the subsequent case in national legal systems. Recent advances in deep learning have enabled a variety of techniques to be used to solve the LJP task.
  arXiv  Detail & Related papers  (2023-10-13T16:47:20Z)
• LAiW: A Chinese Legal Large Language Models Benchmark [17.66376880475554]
  General and legal domain LLMs have demonstrated strong performance in various tasks of LegalAI. We are the first to build the Chinese legal LLMs benchmark LAiW, based on the logic of legal practice.
  arXiv  Detail & Related papers  (2023-10-09T11:19:55Z)
• LawBench: Benchmarking Legal Knowledge of Large Language Models [35.2812008533622]
  Large language models (LLMs) have demonstrated strong capabilities in various aspects. It is unclear how much legal knowledge they possess and whether they can reliably perform legal-related tasks. LawBench has been meticulously crafted to have precise assessment of the LLMs' legal capabilities from three cognitive levels.
  arXiv  Detail & Related papers  (2023-09-28T09:35:59Z)
• A Short Survey of Viewing Large Language Models in Legal Aspect [0.0]
  Large language models (LLMs) have transformed many fields, including natural language processing, computer vision, and reinforcement learning. The integration of LLMs into the legal field has also raised several legal problems, including privacy concerns, bias, and explainability.
  arXiv  Detail & Related papers  (2023-03-16T08:01:22Z)
• Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents [56.40163943394202]
  We release the Longformer-based pre-trained language model, named as Lawformer, for Chinese legal long documents understanding. We evaluate Lawformer on a variety of LegalAI tasks, including judgment prediction, similar case retrieval, legal reading comprehension, and legal question answering.
  arXiv  Detail & Related papers  (2021-05-09T09:39:25Z)
• Authorized and Unauthorized Practices of Law: The Role of Autonomous Levels of AI Legal Reasoning [0.0]
  The legal field has sought to define Authorized Practices of Law (APL) versus Unauthorized Practices of Law (UPL) This paper explores a newly derived instrumental grid depicting the key characteristics underlying APL and UPL as they apply to the AILR autonomous levels.
  arXiv  Detail & Related papers  (2020-08-19T18:35:58Z)
• How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence [81.04070052740596]
  Legal Artificial Intelligence (LegalAI) focuses on applying the technology of artificial intelligence, especially natural language processing, to benefit tasks in the legal domain. This paper introduces the history, the current state, and the future directions of research in LegalAI.
  arXiv  Detail & Related papers  (2020-04-25T14:45:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.