Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models

• URL: http://arxiv.org/abs/2601.03926v1
• Date: Wed, 07 Jan 2026 13:45:39 GMT
• Title: Doc-PP: Document Policy Preservation Benchmark for Large Vision-Language Models
• Authors: Haeun Jang, Hwan Chang, Hwanhee Lee,
• Abstract summary: We introduce Doc-PP, a novel benchmark constructed from real-world reports requiring reasoning across heterogeneous visual and textual elements under strict non-disclosure policies.<n>Our evaluation highlights a systemic Reasoning-Induced Safety Gap: models frequently leak sensitive information when answers must be inferred through complex synthesis or aggregated across modalities.<n>We propose DVA, a structural inference framework that decouples reasoning from policy verification.
• Score: 13.70855540464427
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The deployment of Large Vision-Language Models (LVLMs) for real-world document question answering is often constrained by dynamic, user-defined policies that dictate information disclosure based on context. While ensuring adherence to these explicit constraints is critical, existing safety research primarily focuses on implicit social norms or text-only settings, overlooking the complexities of multimodal documents. In this paper, we introduce Doc-PP (Document Policy Preservation Benchmark), a novel benchmark constructed from real-world reports requiring reasoning across heterogeneous visual and textual elements under strict non-disclosure policies. Our evaluation highlights a systemic Reasoning-Induced Safety Gap: models frequently leak sensitive information when answers must be inferred through complex synthesis or aggregated across modalities, effectively circumventing existing safety constraints. Furthermore, we identify that providing extracted text improves perception but inadvertently facilitates leakage. To address these vulnerabilities, we propose DVA (Decompose-Verify-Aggregation), a structural inference framework that decouples reasoning from policy verification. Experimental results demonstrate that DVA significantly outperforms standard prompting defenses, offering a robust baseline for policy-compliant document understanding

Related papers

• Context Dependence and Reliability in Autoregressive Language Models [4.9988239650406765]
  In critical applications, it is vital to identify which context elements actually influence the output.<n>This work addresses the challenge of distinguishing essential context elements from correlated ones.<n>We introduce RISE, a method that quantifies the unique influence of each input relative to others, minimizing the impact of redundancies.
  arXiv  Detail & Related papers  (2026-02-01T18:25:44Z)
• DeepSynth-Eval: Objectively Evaluating Information Consolidation in Deep Survey Writing [53.85037373860246]
  We introduce Deep Synth-Eval, a benchmark designed to objectively evaluate information consolidation capabilities.<n>We propose a fine-grained evaluation protocol using General Checklists (for factual coverage) and Constraint Checklists (for structural organization)<n>Our results demonstrate that agentic plan-and-write significantly outperform single-turn generation.
  arXiv  Detail & Related papers  (2026-01-07T03:07:52Z)
• Grounding Long-Context Reasoning with Contextual Normalization for Retrieval-Augmented Generation [57.97548022208733]
  We show that seemingly superficial choices in key-value extraction can induce shifts in accuracy and stability.<n>We introduce Contextual Normalization, a strategy that adaptively standardizes context representations before generation.
  arXiv  Detail & Related papers  (2025-10-15T06:28:25Z)
• Towards Reliable Retrieval in RAG Systems for Large Legal Datasets [6.376251215279889]
  Retrieval-Augmented Generation (RAG) is a promising approach to mitigate hallucinations in Large Language Models (LLMs)<n>This is particularly challenging in the legal domain, where large databases of structurally similar documents often cause retrieval systems to fail.<n>We investigate a simple and computationally efficient technique which enhances each text chunk with a document-level synthetic summary.<n>Our work provides evidence that this practical, scalable, and easily integrable technique enhances the reliability of RAG systems when applied to large-scale legal document datasets.
  arXiv  Detail & Related papers  (2025-10-08T13:22:20Z)
• ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search [69.60882125603133]
  We present ReliabilityRAG, a framework for adversarial robustness that explicitly leverages reliability information of retrieved documents.<n>Our work is a significant step towards more effective, provably robust defenses against retrieved corpus corruption in RAG.
  arXiv  Detail & Related papers  (2025-09-27T22:36:42Z)
• Explainable Compliance Detection with Multi-Hop Natural Language Inference on Assurance Case Structure [1.5653612447564105]
  We propose a compliance detection approach based on Natural Language Inference (NLI)<n>We formulate the claim-argument-evidence structure of an assurance case as a multi-hop inference for explainable and traceable compliance detection.<n>Our results highlight the potential of NLI-based approaches in automating the regulatory compliance process.
  arXiv  Detail & Related papers  (2025-06-10T11:56:06Z)
• Keep Security! Benchmarking Security Policy Preservation in Large Language Model Contexts Against Indirect Attacks in Question Answering [15.506092447902931]
  Large Language Models (LLMs) are increasingly deployed in sensitive domains such as enterprise and government.<n>We introduce a novel large-scale benchmark dataset, CoPriva, evaluating LLM adherence to contextual non-disclosure policies in question answering.<n>We evaluate 10 LLMs on our benchmark and reveal a significant vulnerability: many models violate user-defined policies and leak sensitive information.
  arXiv  Detail & Related papers  (2025-05-21T17:58:11Z)
• Advancing Neural Network Verification through Hierarchical Safety Abstract Interpretation [52.626086874715284]
  We introduce a novel problem formulation called Abstract DNN-Verification, which verifies a hierarchical structure of unsafe outputs.<n>By leveraging abstract interpretation and reasoning about output reachable sets, our approach enables assessing multiple safety levels during the formal verification process.<n>Our contributions include a theoretical exploration of the relationship between our novel abstract safety formulation and existing approaches.
  arXiv  Detail & Related papers  (2025-05-08T13:29:46Z)
• Few-shot Policy (de)composition in Conversational Question Answering [54.259440408606515]
  We propose a neuro-symbolic framework to detect policy compliance using large language models (LLMs) in a few-shot setting.<n>We show that our approach soundly reasons about policy compliance conversations by extracting sub-questions to be answered, assigning truth values from contextual information, and explicitly producing a set of logic statements from the given policies.<n>We apply this approach to the popular PCD and conversational machine reading benchmark, ShARC, and show competitive performance with no task-specific finetuning.
  arXiv  Detail & Related papers  (2025-01-20T08:40:15Z)
• Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decoding [118.75567341513897]
  Existing methods typically analyze target text in isolation or solely with non-member contexts.<n>We propose Con-ReCall, a novel approach that leverages the asymmetric distributional shifts induced by member and non-member contexts.
  arXiv  Detail & Related papers  (2024-09-05T09:10:38Z)
• DePrompt: Desensitization and Evaluation of Personal Identifiable Information in Large Language Model Prompts [11.883785681042593]
  DePrompt is a desensitization protection and effectiveness evaluation framework for prompt. We integrate contextual attributes to define privacy types, achieving high-precision PII entity identification. Our framework is adaptable to prompts and can be extended to text usability-dependent scenarios.
  arXiv  Detail & Related papers  (2024-08-16T02:38:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.