InSaAF: Incorporating Safety through Accuracy and Fairness | Are LLMs ready for the Indian Legal Domain?

• URL: http://arxiv.org/abs/2402.10567v4
• Date: Mon, 17 Jun 2024 17:46:07 GMT
• Title: InSaAF: Incorporating Safety through Accuracy and Fairness | Are LLMs ready for the Indian Legal Domain?
• Authors: Yogesh Tripathi, Raghav Donakanti, Sahil Girhepuje, Ishan Kavathekar, Bhaskara Hanuma Vedula, Gokul S Krishnan, Shreya Goyal, Anmol Goel, Balaraman Ravindran, Ponnurangam Kumaraguru,
• Abstract summary: Large Language Models (LLMs) can perform legal tasks in the Indian landscape when social factors are involved. We present a novel metric, $beta$-weighted $textitLegal Safety Score ($LSS_beta$)$, which encapsulates both the fairness and accuracy aspects of the LLM. We propose finetuning pipelines, utilising specialised legal datasets, as a potential method to mitigate bias and improve model safety.
• Score: 18.80574151464707
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in language technology and Artificial Intelligence have resulted in numerous Language Models being proposed to perform various tasks in the legal domain ranging from predicting judgments to generating summaries. Despite their immense potential, these models have been proven to learn and exhibit societal biases and make unfair predictions. In this study, we explore the ability of Large Language Models (LLMs) to perform legal tasks in the Indian landscape when social factors are involved. We present a novel metric, $\beta$-weighted $\textit{Legal Safety Score ($LSS_{\beta}$)}$, which encapsulates both the fairness and accuracy aspects of the LLM. We assess LLMs' safety by considering its performance in the $\textit{Binary Statutory Reasoning}$ task and its fairness exhibition with respect to various axes of disparities in the Indian society. Task performance and fairness scores of LLaMA and LLaMA--2 models indicate that the proposed $LSS_{\beta}$ metric can effectively determine the readiness of a model for safe usage in the legal sector. We also propose finetuning pipelines, utilising specialised legal datasets, as a potential method to mitigate bias and improve model safety. The finetuning procedures on LLaMA and LLaMA--2 models increase the $LSS_{\beta}$, improving their usability in the Indian legal domain. Our code is publicly released.

Related papers

• Multi-Source Retrieval and Reasoning for Legal Sentencing Prediction [50.6851250608938]
  Legal sentencing prediction (LSP) remains difficult due to its need for fine-grained objective knowledge and flexible subjective reasoning.<n>We propose $MSR2$, a framework that integrates multi-source retrieval and reasoning in LLMs with reinforcement learning.<n>Experiments on two real-world datasets show that $MSR2$ improves both accuracy and interpretability in LSP.
  arXiv  Detail & Related papers  (2026-02-04T15:55:55Z)
• SoK: Large Language Model Copyright Auditing via Fingerprinting [69.14570598973195]
  We introduce a unified framework and formal taxonomy that categorizes existing methods into white-box and black-box approaches.<n>We propose LeaFBench, the first systematic benchmark for evaluating LLM fingerprinting under realistic deployment scenarios.
  arXiv  Detail & Related papers  (2025-08-27T12:56:57Z)
• Can LLMs effectively provide game-theoretic-based scenarios for cybersecurity? [51.96049148869987]
  Large Language Models (LLMs) offer new tools and challenges for the security of computer systems.<n>We investigate whether classical game-theoretic frameworks can effectively capture the behaviours of LLM-driven actors and bots.
  arXiv  Detail & Related papers  (2025-08-04T08:57:14Z)
• Can Large Language Models Predict the Outcome of Judicial Decisions? [0.0]
  Large Language Models (LLMs) have shown exceptional capabilities in Natural Language Processing (NLP) We benchmark state-of-the-art open-source LLMs, including LLaMA-3.2-3B and LLaMA-3.1-8B, under varying configurations. Our results demonstrate that fine-tuned smaller models achieve comparable performance to larger models in task-specific contexts.
  arXiv  Detail & Related papers  (2025-01-15T11:32:35Z)
• Adaptive Pruning for Large Language Models with Structural Importance Awareness [66.2690963378878]
  Large language models (LLMs) have significantly improved language understanding and generation capabilities. LLMs are difficult to deploy on resource-constrained edge devices due to their high computational and storage resource demands. We propose structurally-aware adaptive pruning (SAAP) to significantly reduce the computational and memory costs while maintaining model performance.
  arXiv  Detail & Related papers  (2024-12-19T18:08:04Z)
• NyayaAnumana & INLegalLlama: The Largest Indian Legal Judgment Prediction Dataset and Specialized Language Model for Enhanced Decision Analysis [5.790242888372048]
  This paper introduces NyayaAnumana, the largest and most diverse corpus of Indian legal cases compiled for legal judgment prediction (LJP) NyayaAnumana includes a wide range of cases from the Supreme Court, High Courts, Tribunal Courts, District Courts, and Daily Orders. In addition to the dataset, we present INLegalLlama, a domain-specific generative large language model (LLM) tailored to the intricacies of the Indian legal system.
  arXiv  Detail & Related papers  (2024-12-11T13:50:17Z)
• Bayesian scaling laws for in-context learning [72.17734205418502]
  In-context learning (ICL) is a powerful technique for getting language models to perform complex tasks with no training updates. We show that ICL approximates a Bayesian learner and develop a family of novel Bayesian scaling laws for ICL.
  arXiv  Detail & Related papers  (2024-10-21T21:45:22Z)
• LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models [17.90483181611453]
  Large language models (LLMs) have made significant progress in natural language processing tasks and demonstrate considerable potential in the legal domain. Applying existing LLMs to legal systems without careful evaluation of their potential and limitations could pose significant risks in legal practice. We introduce a standardized comprehensive Chinese legal benchmark LexEval.
  arXiv  Detail & Related papers  (2024-09-30T13:44:00Z)
• InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
  InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws. We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries. InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
  arXiv  Detail & Related papers  (2024-06-21T06:19:03Z)
• Temporal Scaling Law for Large Language Models [57.83580734589091]
  We propose the novel concept of Temporal Scaling Law, studying how the test loss of an LLM evolves as the training steps scale up.<n>In contrast to modeling the test loss as a whole in a coarse-grained manner, we break it down and dive into the fine-grained test loss of each token position.<n>We derive the much more precise temporal scaling law by studying the temporal patterns of the parameters in the dynamic hyperbolic-law.
  arXiv  Detail & Related papers  (2024-04-27T05:49:11Z)
• Leveraging open-source models for legal language modeling and analysis: a case study on the Indian constitution [0.0]
  This paper presents a novel approach to legal language modeling (LLM) and analysis using open-source models from Hugging Face. We leverage Hugging Face embeddings via LangChain and Sentence Transformers. We then demonstrate the application of this model by extracting insights from the official Constitution of India.
  arXiv  Detail & Related papers  (2024-04-10T05:35:47Z)
• Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models [107.88745040504887]
  We study the harmlessness alignment problem of multimodal large language models (MLLMs) Inspired by this, we propose a novel jailbreak method named HADES, which hides and amplifies the harmfulness of the malicious intent within the text input. Experimental results show that HADES can effectively jailbreak existing MLLMs, which achieves an average Attack Success Rate (ASR) of 90.26% for LLaVA-1.5 and 71.60% for Gemini Pro Vision.
  arXiv  Detail & Related papers  (2024-03-14T18:24:55Z)
• She had Cobalt Blue Eyes: Prompt Testing to Create Aligned and Sustainable Language Models [2.6089354079273512]
  Recent events indicate ethical concerns around conventionally trained large language models (LLMs) We introduce a test suite of prompts to foster the development of aligned LLMs that are fair, safe, and robust. Our test suite evaluates outputs from four state-of-the-art language models: GPT-3.5, GPT-4, OPT, and LLaMA-2.
  arXiv  Detail & Related papers  (2023-10-20T14:18:40Z)
• Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration [52.57055162778548]
  Legal Judgment Prediction (LJP) has become an increasingly crucial task in Legal AI. Precedents are the previous legal cases with similar facts, which are the basis for the judgment of the subsequent case in national legal systems. Recent advances in deep learning have enabled a variety of techniques to be used to solve the LJP task.
  arXiv  Detail & Related papers  (2023-10-13T16:47:20Z)
• SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore [159.21914121143885]
  We present SILO, a new language model that manages this risk-performance tradeoff during inference. SILO is built by (1) training a parametric LM on Open License Corpus (OLC), a new corpus we curate with 228B tokens of public domain and permissively licensed text. Access to the datastore greatly improves out of domain performance, closing 90% of the performance gap with an LM trained on the Pile.
  arXiv  Detail & Related papers  (2023-08-08T17:58:15Z)
• Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence [5.07013500385659]
  This paper explores Large Language Models' (LLMs) capabilities in applying tax law. Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release. Findings indicate that LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy but not yet at expert tax lawyer levels.
  arXiv  Detail & Related papers  (2023-06-12T12:40:48Z)
• Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents [56.40163943394202]
  We release the Longformer-based pre-trained language model, named as Lawformer, for Chinese legal long documents understanding. We evaluate Lawformer on a variety of LegalAI tasks, including judgment prediction, similar case retrieval, legal reading comprehension, and legal question answering.
  arXiv  Detail & Related papers  (2021-05-09T09:39:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.