Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows

• URL: http://arxiv.org/abs/2607.27648v1
• Date: Thu, 30 Jul 2026 03:59:42 GMT
• Title: Not as Sweet by Another Name: An Empirical Study of Format Robustness in LLM Document Workflows
• Abstract summary: We conduct a large-scale empirical study spanning four representative tasks, four real-world tasks, and four document formats.<n>Our findings reveal that format variation poses a systematic and serious threat.<n>Our study demonstrates that document format is not a neutral wrapper but a critical factor affecting the reliability of LLM systems.
• Score: 19.259315558139573
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-driven software systems are rapidly evolving from plain-text conversations to document-centric end-to-end workflows, where the same semantic content can be delivered in diverse document formats (e.g., CSV) through file upload interfaces. Yet existing testing work focuses on the robustness and reliability of models and systems whose input is a single prompt string, leaving a critical question unanswered: Can these document workflows maintain robust behaviors when the same content arrives in a different document format? To fill the gap, in this paper, we propose a format-aware metamorphic testing framework with three metamorphic relations to comprehensively evaluate the format robustness of end-to-end LLM document workflows. Based on this framework, we conduct a large-scale empirical study spanning four representative LLM workflows, four real-world tasks, and four document formats, comprising a total of 48,000 workflow executions. Our findings reveal that format variation poses a systematic and serious threat. Merely switching formats can cause accuracy to drop by up to 53.63% and trigger decision drifts in over 41% of instances. We further design lightweight mitigation strategies from the users' perspective that recover up to 44.21% of format-induced decision drift without model retraining. Our study demonstrates that document format is not a neutral wrapper but a critical factor affecting the reliability of LLM software systems, calling for corresponding testing and safeguards in the deployment in real-world high-stakes scenarios.

Related papers

• What to Format and How: A Benchmark and Workflow Approach for Document Formatting [25.500100602233076]
  Real-world formatting often requires identifying targets based on document content.<n>We introduce DocFormBench, a benchmark that extends Text-to-Format evaluation to diverse formats.<n>We propose DocFormFlow, a method that decouples target localization from modification execution into what to format and how.
  arXiv  Detail & Related papers  (2026-06-01T09:02:33Z)
• Hybrid OCR-LLM Framework for Enterprise-Scale Document Information Extraction Under Copy-heavy Task [11.672798725644121]
  This work strategically combines OCR engines with Large Language Models (LLMs) to optimize the accuracy-efficiency trade-off inherent in repetitive document extraction tasks.<n>We implement and evaluate 25 configurations across three extraction paradigms (direct, replacement, and table-based) on identity documents spanning four formats.
  arXiv  Detail & Related papers  (2025-10-11T09:40:34Z)
• On-Premise AI for the Newsroom: Evaluating Small Language Models for Investigative Document Search [2.853035319109148]
  Large language models (LLMs) with retrieval-augmented generation (RAG) capabilities promise to accelerate the process of document discovery.<n>We present a journalist-centered approach to search that prioritizes transparency and editorial control through a five-stage pipeline.<n>We evaluate three quantized models (Gemma 3 12B, Qwen 3 14B, and GPT-OSS 20B) on two corpora and find substantial variation in reliability.
  arXiv  Detail & Related papers  (2025-09-29T20:50:40Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• On the Comprehensibility of Multi-structured Financial Documents using LLMs and Pre-processing Tools [41.122196205034165]
  This paper explores the capabilities of Large Language Models (LLMs) and Multi-modal Large Language Models (MLLMs) in understanding and answering questions from complex data structures found in PDF documents.<n>Our findings indicate that GPT-4o, a popular MLLM, achieves an accuracy of 56% on multi-structured documents when fed documents directly.
  arXiv  Detail & Related papers  (2025-06-05T15:52:44Z)
• WildDoc: How Far Are We from Achieving Comprehensive and Robust Document Understanding in the Wild? [64.62909376834601]
  This paper introduces WildDoc, the inaugural benchmark designed specifically for assessing document understanding in natural environments.<n> evaluation of state-of-the-art MLLMs on WildDoc expose substantial performance declines and underscore the models' inadequate robustness compared to traditional benchmarks.
  arXiv  Detail & Related papers  (2025-05-16T09:09:46Z)
• M-DocSum: Do LVLMs Genuinely Comprehend Interleaved Image-Text in Document Summarization? [49.53982792497275]
  We investigate whether Large Vision-Language Models (LVLMs) genuinely comprehend interleaved image-text in the document.<n>Existing document understanding benchmarks often assess LVLMs using question-answer formats.<n>We introduce a novel and challenging Multimodal Document Summarization Benchmark (M-DocSum-Bench)<n>M-DocSum-Bench comprises 500 high-quality arXiv papers, along with interleaved multimodal summaries aligned with human preferences.
  arXiv  Detail & Related papers  (2025-03-27T07:28:32Z)
• LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs [69.40865293066885]
  We present the first systematic evaluation examining format bias in performance of large language models (LLMs)<n>We present our empirical format bias evaluation spanning four commonly used categories -- multiple-choice question-answer, wrapping, list, and mapping.
  arXiv  Detail & Related papers  (2024-08-16T10:45:45Z)
• DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems [99.17123445211115]
  We introduce DocBench, a benchmark to evaluate large language model (LLM)-based document reading systems. Our benchmark involves the recruitment of human annotators and the generation of synthetic questions. It includes 229 real documents and 1,102 questions, spanning across five different domains and four major types of questions.
  arXiv  Detail & Related papers  (2024-07-15T13:17:42Z)
• Robustness of Structured Data Extraction from In-plane Rotated Documents using Multi-Modal Large Language Models (LLM) [0.0]
  This study investigates the impact of document skew on the data extraction accuracy of three state-of-the-art multi-modal models. We identify the safe in-plane rotation angles (SIPRA) for each model and investigate the effects of skew on model hallucinations.
  arXiv  Detail & Related papers  (2024-06-13T08:55:01Z)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting [68.19544657508509]
  Large language models (LLMs) are adopted as a fundamental component of language technologies. We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt format in few-shot settings. We propose an algorithm that rapidly evaluates a sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights.
  arXiv  Detail & Related papers  (2023-10-17T15:03:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.