AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System

• URL: http://arxiv.org/abs/2607.18825v1
• Date: Tue, 21 Jul 2026 08:01:17 GMT
• Title: AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System
• Abstract summary: This study introduces an advanced Artificial Intelligence for Indian Legal Question Answering (AILQA) system tailored to the Indian legal context.<n>AILQA leverages a variety of embedding and generative models, including recent Large Language Models (LLMs)<n>We conducted rigorous evaluations using both lexical and semantic metrics, enriched by expert legal feedback, to ensure relevance and accuracy.
• Score: 11.383388187292674
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This comprehensive study introduces an advanced Artificial Intelligence for Indian Legal Question Answering (AILQA) system tailored to the Indian legal context. AILQA leverages a variety of embedding and generative models, including recent Large Language Models (LLMs), to address the unique challenges posed by the intricate and diverse nature of Indian legal texts and to enhance the accuracy and reliability of responses to legal questions. We conducted rigorous evaluations using both lexical and semantic metrics, enriched by expert legal feedback, to ensure relevance and accuracy. Our findings underscore the effectiveness of the Retrieval-Augmented Generation (RAG) paradigm in improving answer quality, particularly in complex legal domains. Additionally, we assessed performance on standardized tests such as the All India Bar Examination (AIBE), thereby providing a robust benchmark for practical applications. Under the study's evaluation protocol, some AI-generated responses received higher ratings than the available reference answers, particularly when they contained accurate and relevant supporting details. This finding is specific to the evaluated dataset and rating criteria and should not be interpreted as evidence that the models generally outperform qualified legal professionals. We also discuss the challenges encountered, such as the need for precise context and the risks of model hallucination, and propose directions for future research to further refine AI capabilities in the legal field. This study aims to pave the way for enhanced legal decision-support systems, making them more accessible and effective for legal professionals and the public alike.

Related papers

• Rhetorical-Role-Aware Retrieval-Augmented Generation for Legal Question Answering over Indian Supreme Court Judgments [0.4893345190925178]
  This research paper proposes a Retrieval Augmented Generation (RAG) framework that is specific to the legal field in order to assist interactive retrieval and reason about judgments from the Supreme Court of India.<n>The solution uses an enhanced version of RAG framework which consists of rhetorically based chunking, fusion-based retrieval, and cross encoder reranking methods to increase the relevancy of the information retrieved.
  arXiv  Detail & Related papers  (2026-08-07T05:38:02Z)
• WisdomInterrogatory (LuWen): An Open-Source Legal Large Language Model Technical Report [55.27414605169639]
  We present WisdomInterrogatory (LuWen), an open-source Chinese legal language model built upon the Baichuan foundation model.<n>We evaluate LuWen on five representative legal tasks spanning both prediction and generation settings.
  arXiv  Detail & Related papers  (2026-04-08T06:59:07Z)
• LegalOne: A Family of Foundation Models for Reliable Legal Reasoning [54.57434222018289]
  We present LegalOne, a family of foundational models specifically tailored for the Chinese legal domain.<n>LegalOne is developed through a comprehensive three-phase pipeline designed to master legal reasoning.<n>We publicly release the LegalOne weights and the LegalKit evaluation framework to advance the field of Legal AI.
  arXiv  Detail & Related papers  (2026-01-31T10:18:32Z)
• ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India [10.522785783474857]
  We introduce Reinforcement Learning-based Legal Reasoning (ReGal), a framework that integrates Multi-Task Instruction Tuning with Reinforcement Learning from AI Feedback.<n>Our approach is evaluated across two critical legal tasks: (i) Court Judgment Prediction and Explanation (CJPE), and (ii) Legal Document Summarization.
  arXiv  Detail & Related papers  (2025-12-19T19:13:41Z)
• Assessing the Reliability of Large Language Models in the Bengali Legal Context: A Comparative Evaluation Using LLM-as-Judge and Legal Experts [0.0]
  Generative AI models like OpenAI GPT-4.1 Mini, Gemini 2.0 Flash, Meta Llama 3 70B, and DeepSeek R1 could potentially democratize legal assistance.<n>In this study, we collected 250 authentic legal questions from the Facebook group "Know Your Rights"<n>We evaluated each AI-generated response across four critical dimensions: factual accuracy, legal appropriateness, completeness, and clarity.
  arXiv  Detail & Related papers  (2025-11-07T02:44:00Z)
• Judicial Requirements for Generative AI in Legal Reasoning [0.0]
  Large Language Models (LLMs) are being integrated into professional domains, yet their limitations in high-stakes fields like law remain poorly understood.<n>This paper defines the core capabilities that an AI system must possess to function as a reliable reasoning tool in judicial decision-making.
  arXiv  Detail & Related papers  (2025-08-26T09:56:26Z)
• ClaimGen-CN: A Large-scale Chinese Dataset for Legal Claim Generation [56.79698529022327]
  Legal claims refer to the plaintiff's demands in a case and are essential to guiding judicial reasoning and case resolution.<n>This paper explores the problem of legal claim generation based on the given case's facts.<n>We construct ClaimGen-CN, the first dataset for Chinese legal claim generation task.
  arXiv  Detail & Related papers  (2025-08-24T07:19:25Z)
• GLARE: Agentic Reasoning for Legal Judgment Prediction [60.13483016810707]
  Legal judgment prediction (LJP) has become increasingly important in the legal field.<n>Existing large language models (LLMs) have significant problems of insufficient reasoning due to a lack of legal knowledge.<n>We introduce GLARE, an agentic legal reasoning framework that dynamically acquires key legal knowledge by invoking different modules.
  arXiv  Detail & Related papers  (2025-08-22T13:38:12Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Legal Question-Answering in the Indian Context: Efficacy, Challenges, and Potential of Modern AI Models [3.552993426200889]
  Legal QA platforms bear the promise to metamorphose the manner in which legal experts engage with jurisprudential documents. Our discourse zeroes in on an array of retrieval and QA mechanisms, positioning the OpenAI GPT model as a reference point. The ambit of this study is tethered to the Indian criminal legal landscape, distinguished by its intricate nature and associated logistical constraints.
  arXiv  Detail & Related papers  (2023-09-26T07:56:55Z)
• ExpertQA: Expert-Curated Questions and Attributed Answers [51.68314045809179]
  We conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality. We collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.
  arXiv  Detail & Related papers  (2023-09-14T16:54:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.