From Policy to Logic for Efficient and Interpretable Coverage Assessment

• URL: http://arxiv.org/abs/2601.01266v2
• Date: Thu, 08 Jan 2026 18:28:40 GMT
• Title: From Policy to Logic for Efficient and Interpretable Coverage Assessment
• Abstract summary: We present an approach designed to support human reviewers by making policy interpretation more efficient and interpretable.<n>We introduce a methodology that pairs a coverage-aware retriever with symbolic rule-based reasoning to surface relevant policy language.<n>Our approach achieves a 44% reduction in inference cost alongside a 4.5% improvement in F1 score, demonstrating both efficiency and effectiveness.
• Score: 4.195679119463805
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have demonstrated strong capabilities in interpreting lengthy, complex legal and policy language. However, their reliability can be undermined by hallucinations and inconsistencies, particularly when analyzing subjective and nuanced documents. These challenges are especially critical in medical coverage policy review, where human experts must be able to rely on accurate information. In this paper, we present an approach designed to support human reviewers by making policy interpretation more efficient and interpretable. We introduce a methodology that pairs a coverage-aware retriever with symbolic rule-based reasoning to surface relevant policy language, organize it into explicit facts and rules, and generate auditable rationales. This hybrid system minimizes the number of LLM inferences required which reduces overall model cost. Notably, our approach achieves a 44% reduction in inference cost alongside a 4.5% improvement in F1 score, demonstrating both efficiency and effectiveness.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Summarization is Not Dead Yet [28.302567995407532]
  The progress of large language models (LLMs) has fueled claims that model-generated summaries rival or even surpass human-written references.<n>We re-examine this narrative through a multi-track evaluation covering five diverse datasets and five state-of-the-art LLMs.<n>Our findings reveal a more nuanced landscape in which human reference summaries continue to demonstrate advantages in informativeness and faithfulness.
  arXiv  Detail & Related papers  (2026-06-06T06:38:35Z)
• From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text [0.0]
  This paper introduces a comprehensive dual-aspect evaluation framework to address this need.<n>We establish a performance benchmark for four state-of-the-art large language models across three key dimensions: Accuracy, Readability, and Consistency.<n>Our results reveal a crucial trade-off: models like Grok-1 excel in Readability and Consistency but compromise on fine-grained legal Accuracy.
  arXiv  Detail & Related papers  (2026-04-17T17:28:23Z)
• Perception-Consistency Multimodal Large Language Models Reasoning via Caption-Regularized Policy Optimization [72.30168853571216]
  multimodal large language models excel at tasks that integrate visual perception with symbolic reasoning.<n>CapPO integrates two key mechanisms: (1) a caption-based consistency regularization, which minimizes the divergence between responses conditioned on raw images and those conditioned on captions, and (2) a KL-weighted advantage estimation scheme, which adaptively scales reinforcement signals to strengthen perceptually consistent trajectories.
  arXiv  Detail & Related papers  (2025-09-26T04:32:26Z)
• DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understanding [66.07724324530844]
  We propose DocThinker, a rule-based Reinforcement Learning framework for dynamic inference-time reasoning.<n>Our method mitigates catastrophic forgetting and enhances both adaptability and transparency.<n>Our findings highlight RL as a powerful alternative for enhancing explainability and adaptability in MLLM-based document understanding.
  arXiv  Detail & Related papers  (2025-08-12T03:06:55Z)
• Assessing the Reliability and Validity of Large Language Models for Automated Assessment of Student Essays in Higher Education [0.30158609733245967]
  Five advanced Large Language Models (LLMs), Claude 3.5, DeepSeek v2, Gemini 2.5, GPT-4, and Mistral 24B, were investigated for automated essay scoring in a higher education context.<n>A total of 67 Italian-language student essays were evaluated using a four-criterion rubric.<n>Human-LLM agreement was consistently low and non-significant, and within-model reliability across replications was similarly weak.
  arXiv  Detail & Related papers  (2025-08-04T14:02:12Z)
• Identifying Algorithmic and Domain-Specific Bias in Parliamentary Debate Summarisation [13.982513887272683]
  This paper explores the use of large language models (LLMs) to summarise plenary debates from the European Parliament.<n>We propose a structured, multi-stage summarisation framework that improves textual coherence and content fidelity.<n>We find evidence of consistent positional and partisan biases, with certain speakers systematically under-represented or misattributed.
  arXiv  Detail & Related papers  (2025-07-16T11:49:33Z)
• Faithful and Robust LLM-Driven Theorem Proving for NLI Explanations [13.485604499678262]
  Natural language explanations play a fundamental role in Natural Language Inference (NLI)<n>Recent work has shown that the interaction of large language models (LLMs) with theorem provers (TPs) can help verify and improve the validity of NLI explanations.<n>This paper investigates strategies to alleviate semantic loss during autoformalisation.
  arXiv  Detail & Related papers  (2025-05-30T06:38:39Z)
• PlainQAFact: Retrieval-augmented Factual Consistency Evaluation Metric for Biomedical Plain Language Summarization [5.5899921245557]
  Hallucinated outputs from large language models pose risks in the medical domain.<n>We introduce PlainQAFact, an automatic factual consistency evaluation metric trained on a fine-grained, human-annotated dataset PlainFact.
  arXiv  Detail & Related papers  (2025-03-11T20:59:53Z)
• LLM-Safety Evaluations Lack Robustness [58.334290876531036]
  We argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise.<n>We propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers.
  arXiv  Detail & Related papers  (2025-03-04T12:55:07Z)
• LlaMADRS: Prompting Large Language Models for Interview-Based Depression Assessment [75.44934940580112]
  This study introduces LlaMADRS, a novel framework leveraging open-source Large Language Models (LLMs) to automate depression severity assessment.<n>We employ a zero-shot prompting strategy with carefully designed cues to guide the model in interpreting and scoring transcribed clinical interviews.<n>Our approach, tested on 236 real-world interviews, demonstrates strong correlations with clinician assessments.
  arXiv  Detail & Related papers  (2025-01-07T08:49:04Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation [60.65820977963331]
  We introduce a novel evaluation paradigm for Large Language Models (LLMs) This paradigm shifts the emphasis from result-oriented assessments, which often neglect the reasoning process, to a more comprehensive evaluation. By applying this paradigm in the GSM8K dataset, we have developed the MR-GSM8K benchmark.
  arXiv  Detail & Related papers  (2023-12-28T15:49:43Z)
• From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews [14.135107583299277]
  This study explores the integration of Large Language Models (LLMs) with human expertise to enhance text analysis of stakeholder interviews regarding K-12 education policy within one U.S. state. Using a mixed-methods approach, human experts developed a codebook and coding processes as informed by domain knowledge and unsupervised topic modeling results. Results reveal that while GPT-4 thematic coding aligned with human coding by 77.89% at specific themes, expanding to broader themes increased congruence to 96.02%, surpassing traditional Natural Language Processing (NLP) methods by over 25%.
  arXiv  Detail & Related papers  (2023-12-02T18:55:14Z)
• Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization [66.08074487429477]
  We investigate the stability and reliability of large language models (LLMs) as automatic evaluators for abstractive summarization. We find that while ChatGPT and GPT-4 outperform the commonly used automatic metrics, they are not ready as human replacements.
  arXiv  Detail & Related papers  (2023-05-22T14:58:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.