Blind Judgement: Agent-Based Supreme Court Modelling With GPT

• URL: http://arxiv.org/abs/2301.05327v1
• Date: Thu, 12 Jan 2023 23:07:55 GMT
• Title: Blind Judgement: Agent-Based Supreme Court Modelling With GPT
• Abstract summary: We train nine separate models with the respective authored opinions of each supreme justice active ca. 2015 and test the resulting system on 96 real-world cases. We find our system predicts the decisions of the real-world Supreme Court with better-than-random accuracy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present a novel Transformer-based multi-agent system for simulating the judicial rulings of the 2010-2016 Supreme Court of the United States. We train nine separate models with the respective authored opinions of each supreme justice active ca. 2015 and test the resulting system on 96 real-world cases. We find our system predicts the decisions of the real-world Supreme Court with better-than-random accuracy. We further find a correlation between model accuracy with respect to individual justices and their alignment between legal conservatism & liberalism. Our methods and results hold significance for researchers interested in using language models to simulate politically-charged discourse between multiple agents.

Related papers

• CyberJurors: A Multi-Agent Simulation Task for E-Commerce Disputes Verdict [59.06704011432714]
  E-commerce platforms have begun recruiting crowdsourced jurors to adjudicate massive volumes of transaction disputes.<n>We introduce a pioneering task, E-commerce Dispute Verdicts (EDV), and present VerdictBench, a multimodal benchmark comprising 6,000 real-world cases.<n>We propose CyberJurors, a multi-agent framework to clarify the dispute logic and regulate the verdict process.
  arXiv  Detail & Related papers  (2026-05-27T12:07:44Z)
• Multi-Agent Debate for LLM Judges with Adaptive Stability Detection [46.67172123607961]
  We propose a multi-agent debate judge framework where agents collaboratively reason and iteratively refine their responses.<n>We formalize the debate process mathematically, analyzing agent interactions and proving that debate amplifies correctness compared to static ensembles.<n> Experiments across multiple benchmarks and models demonstrate that our framework improves judgment accuracy over majority voting while maintaining computational efficiency.
  arXiv  Detail & Related papers  (2025-10-14T16:30:30Z)
• The Judge Variable: Challenging Judge-Agnostic Legal Judgment Prediction [0.0]
  This study examines the role of human judges in legal decision-making.<n>It uses machine learning to predict child physical custody outcomes in French appellate courts.
  arXiv  Detail & Related papers  (2025-07-18T08:28:53Z)
• Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models [66.51871176061195]
  Decentralized Arena (dearena) is a fully automated framework leveraging collective intelligence from all large language models to evaluate each other.<n> dearena attains up to 97% correlation with human judgements, while significantly reducing the cost.
  arXiv  Detail & Related papers  (2025-05-19T07:34:25Z)
• Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models [68.92020689188887]
  Retrieval-Augmented Generation (RAG) has proven its effectiveness in alleviating hallucinations for Large Language Models (LLMs) Existing automated evaluation metrics cannot fairly evaluate the outputs generated by RAG models during training and evaluation. This paper introduces the Judge-Consistency (ConsJudge) method, which aims to enhance LLMs to generate more accurate evaluations for RAG models.
  arXiv  Detail & Related papers  (2025-02-26T04:50:43Z)
• Fact-or-Fair: A Checklist for Behavioral Testing of AI Models on Fairness-Related Queries [85.909363478929]
  In this study, we focus on 19 real-world statistics collected from authoritative sources. We develop a checklist comprising objective and subjective queries to analyze behavior of large language models. We propose metrics to assess factuality and fairness, and formally prove the inherent trade-off between these two aspects.
  arXiv  Detail & Related papers  (2025-02-09T10:54:11Z)
• Embedding-based Approaches to Hyperpartisan News Detection [6.522338519818378]
  Hyperpartisan news is news that takes an extremely polarized political standpoint with an intention of creating political divide among the public. Our best system using pre-trained ELMo with Bidirectional LSTM achieved an accuracy of 83% through 10-fold cross-validation.
  arXiv  Detail & Related papers  (2025-01-02T17:29:53Z)
• ElectionSim: Massive Population Election Simulation Powered by Large Language Model Driven Agents [70.17229548653852]
  We introduce ElectionSim, an innovative election simulation framework based on large language models. We present a million-level voter pool sampled from social media platforms to support accurate individual simulation. We also introduce PPE, a poll-based presidential election benchmark to assess the performance of our framework under the U.S. presidential election scenario.
  arXiv  Detail & Related papers  (2024-10-28T05:25:50Z)
• Representation Bias in Political Sample Simulations with Large Language Models [54.48283690603358]
  This study seeks to identify and quantify biases in simulating political samples with Large Language Models. Using the GPT-3.5-Turbo model, we leverage data from the American National Election Studies, German Longitudinal Election Study, Zuobiao dataset, and China Family Panel Studies.
  arXiv  Detail & Related papers  (2024-07-16T05:52:26Z)
• Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges [6.609843448260634]
  The LLM-as-a-judge paradigm is rapidly gaining traction as an approach to evaluating large language models. This paper focuses on a clean scenario in which inter-human agreement is high. We identify vulnerabilities in judge models, such as their sensitivity to prompt complexity and length, and a tendency toward leniency.
  arXiv  Detail & Related papers  (2024-06-18T13:49:54Z)
• Towards Explainability in Legal Outcome Prediction Models [64.00172507827499]
  We argue that precedent is a natural way of facilitating explainability for legal NLP models. By developing a taxonomy of legal precedent, we are able to compare human judges and neural models. We find that while the models learn to predict outcomes reasonably well, their use of precedent is unlike that of human judges.
  arXiv  Detail & Related papers  (2024-03-25T15:15:41Z)
• Aligning Large Language Models by On-Policy Self-Judgment [49.31895979525054]
  Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model (RM) for on-policy learning. We present a novel alignment framework, SELF-JUDGE, that does on-policy learning and is parameter efficient. We show that the rejecting sampling by itself can improve performance further without an additional evaluator.
  arXiv  Detail & Related papers  (2024-02-17T11:25:26Z)
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges [54.007823006976516]
  We propose to fine-tune Large Language Models (LLMs) as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks. We first propose a comprehensive, large-scale, high-quality dataset containing task seeds, LLMs-generated answers, and GPT-4-generated judgments for fine-tuning high-performance judges. We then analyze the key biases in fine-tuning LLM as a judge and consider them as position bias, knowledge bias, and format bias.
  arXiv  Detail & Related papers  (2023-10-26T17:48:58Z)
• Legal and Political Stance Detection of SCOTUS Language [9.906317924458456]
  We analyze publicly available US Supreme Court documents using automated stance detection. We calculate two distinct ideology metrics of SCOTUS justices using oral argument transcripts. We find that justices who are more responsive to public opinion tend to express their ideology during oral arguments.
  arXiv  Detail & Related papers  (2022-11-21T18:45:57Z)
• Accuracy, Fairness, and Interpretability of Machine Learning Criminal Recidivism Models [4.297070083645049]
  Various machine learning-based criminal recidivism models are created based on a real-world parole decision dataset from the state of Georgia in the United States. It is found that there are noted differences and trade-offs between accuracy, fairness, and being inherently interpretable.
  arXiv  Detail & Related papers  (2022-09-14T17:53:24Z)
• JUSTICE: A Benchmark Dataset for Supreme Court's Judgment Prediction [0.0]
  We aim to create a high-quality dataset of SCOTUS court cases so that they may be readily used in natural language processing (NLP) research and other data-driven applications. By using advanced NLP algorithms to analyze previous court cases, the trained models are able to predict and classify a court's judgment.
  arXiv  Detail & Related papers  (2021-12-06T23:19:08Z)
• ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation [3.285073688021526]
  We propose the task of Court Judgment Prediction and Explanation (CJPE) CJPE requires an automated system to predict an explainable outcome of a case. Our best prediction model has an accuracy of 78% versus 94% for human legal experts.
  arXiv  Detail & Related papers  (2021-05-28T03:07:32Z)
• Equality before the Law: Legal Judgment Consistency Analysis for Fairness [55.91612739713396]
  In this paper, we propose an evaluation metric for judgment inconsistency, Legal Inconsistency Coefficient (LInCo) We simulate judges from different groups with legal judgment prediction (LJP) models and measure the judicial inconsistency with the disagreement of the judgment results given by LJP models trained on different groups. We employ LInCo to explore the inconsistency in real cases and come to the following observations: (1) Both regional and gender inconsistency exist in the legal system, but gender inconsistency is much less than regional inconsistency.
  arXiv  Detail & Related papers  (2021-03-25T14:28:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.