PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models

• URL: http://arxiv.org/abs/2604.12995v1
• Date: Tue, 14 Apr 2026 17:27:50 GMT
• Title: PolicyLLM: Towards Excellent Comprehension of Public Policy for Large Language Models
• Abstract summary: Large Language Models (LLMs) are increasingly integrated into real-world decision-making.<n>Yet, their ability to comprehend and reason about policy-related content remains underexplored.<n>We present textbftextitPolicyBench, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension.
• Score: 56.37206340175191
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly integrated into real-world decision-making, including in the domain of public policy. Yet, their ability to comprehend and reason about policy-related content remains underexplored. To fill this gap, we present \textbf{\textit{PolicyBench}}, the first large-scale cross-system benchmark (US-China) evaluating policy comprehension, comprising 21K cases across a broad spectrum of policy areas, capturing the diversity and complexity of real-world governance. Following Bloom's taxonomy, the benchmark assesses three core capabilities: (1) \textbf{Memorization}: factual recall of policy knowledge, (2) \textbf{Understanding}: conceptual and contextual reasoning, and (3) \textbf{Application}: problem-solving in real-life policy scenarios. Building on this benchmark, we further propose \textbf{\textit{PolicyMoE}}, a domain-specialized Mixture-of-Experts (MoE) model with expert modules aligned to each cognitive level. The proposed models demonstrate stronger performance on application-oriented policy tasks than on memorization or conceptual understanding, and yields the highest accuracy on structured reasoning tasks. Our results reveal key limitations of current LLMs in policy understanding and suggest paths toward more reliable, policy-focused models.

Related papers

• PolicyMem: Geometric Policy Memory for LLM Governance [72.04962480384322]
  We introduce PolicyMem, a geometric policy memory that externalizes natural-language policies as reusable geometric memory objects.<n>A memory writer compiles natural-language policies into policy memory slots, and query-response pairs read the policy memory through projection energy.<n>The resulting policy-evidence profile directly mediates the safety verdict and is reused for policy attribution and post-intervention verification.
  arXiv  Detail & Related papers  (2026-09-12T06:07:43Z)
• Learning Fair Pareto-Optimal Policies in Multi-Objective Reinforcement Learning [1.3483705601540006]
  We formalize the fair optimization problem in multi-policy MORL.<n>We show that for concave, piecewise-linear welfare functions, fair policies remain in the convex coverage set.<n>We demonstrate that non-stationary policies, augmented with accrued reward histories, and state policies improve fairness by dynamically adapting to historical inequities.
  arXiv  Detail & Related papers  (2026-06-16T16:16:54Z)
• DRIP-R: A Benchmark for Decision-Making and Reasoning Under Real-World Policy Ambiguity in the Retail Domain [21.9900052246381]
  We introduce DRIPR, a benchmark that exploits realworld retail policy construct scenarios in which no single correct resolution exists.<n>DRIPR comprises a curated set of policyambiguous return scenarios paired with a realistic customer personas, a full conversational simulation tool-calling capabilities and a multi-judge evaluation framework.<n>Our experiments show that frontier models fundamentally disagree on identical policy ambiguities, confirming that ambiguity poses a genuine and systematic challenge to LLM decisionmaking.
  arXiv  Detail & Related papers  (2026-05-08T13:10:49Z)
• Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models [13.70855540464427]
  We introduce Doc-PP, a novel benchmark constructed from real-world reports requiring reasoning across heterogeneous visual and textual elements under strict non-disclosure policies.<n>Our evaluation highlights a systemic Reasoning-Induced Safety Gap: models frequently leak sensitive information when answers must be inferred through complex synthesis or aggregated across modalities.<n>We propose DVA, a structural inference framework that decouples reasoning from policy verification.
  arXiv  Detail & Related papers  (2026-01-07T13:45:39Z)
• Multimodal Policy Internalization for Conversational Agents [48.11601444262434]
  Multimodal Policy Internalization (MPI) is a new task that internalizes reasoning-intensive multimodal policies into model parameters.<n>We build two datasets spanning synthetic and real-world decision-making and tool-using tasks.<n>TriMPI achieves notable gains in end-to-end accuracy, generalization, and robustness.
  arXiv  Detail & Related papers  (2025-10-10T15:28:30Z)
• Convergence and Sample Complexity of First-Order Methods for Agnostic Reinforcement Learning [66.4260157478436]
  We study reinforcement learning in the policy learning setting.<n>The goal is to find a policy whose performance is competitive with the best policy in a given class of interest.
  arXiv  Detail & Related papers  (2025-07-06T14:40:05Z)
• Reinforcement Learning with Continuous Actions Under Unmeasured Confounding [14.510042451844766]
  This paper addresses the challenge of offline policy learning in reinforcement learning with continuous action spaces.<n>We develop a minimax estimator and introduce a policy-gradient-based algorithm to identify the in-class optimal policy.<n>We provide theoretical results regarding the consistency, finite-sample error bound, and regret bound of the resulting optimal policy.
  arXiv  Detail & Related papers  (2025-05-01T04:55:29Z)
• Policy Regularization on Globally Accessible States in Cross-Dynamics Reinforcement Learning [53.9544543607396]
  We propose a novel framework that integrates reward rendering with Imitation from Observation (IfO)<n>By instantiating F-distance in different ways, we derive two theoretical analysis and develop a practical algorithm called Accessible State Oriented Policy Regularization (ASOR)<n>ASOR serves as a general add-on module that can be incorporated into various approaches RL, including offline RL and off-policy RL.
  arXiv  Detail & Related papers  (2025-03-10T03:50:20Z)
• From Explainability to Interpretability: Interpretable Policies in Reinforcement Learning Via Model Explanation [2.08099858257632]
  We propose a novel model-agnostic approach to transform complex deep RL policies into transparent representations.<n>We evaluate our approach with three existing deep RL algorithms and validate its performance in two classic control environments.
  arXiv  Detail & Related papers  (2025-01-16T22:11:03Z)
• Towards A Unified Policy Abstraction Theory and Representation Learning Approach in Markov Decision Processes [39.94472154078338]
  We propose a unified policy abstraction theory, containing three types of policy abstraction associated to policy features at different levels. We then generalize them to three policy metrics that quantify the distance (i.e., similarity) of policies. For the empirical study, we investigate the efficacy of the proposed policy metrics and representations, in characterizing policy difference and conveying policy generalization respectively.
  arXiv  Detail & Related papers  (2022-09-16T03:41:50Z)
• Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients [54.98496284653234]
  We consider the task of training a policy that maximizes reward while minimizing disclosure of certain sensitive state variables through the actions. We solve this problem by introducing a regularizer based on the mutual information between the sensitive state and the actions. We develop a model-based estimator for optimization of privacy-constrained policies.
  arXiv  Detail & Related papers  (2020-12-30T03:22:35Z)
• Policy Evaluation Networks [50.53250641051648]
  We introduce a scalable, differentiable fingerprinting mechanism that retains essential policy information in a concise embedding. Our empirical results demonstrate that combining these three elements can produce policies that outperform those that generated the training data.
  arXiv  Detail & Related papers  (2020-02-26T23:00:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.