The Format Tax

• URL: http://arxiv.org/abs/2604.03616v1
• Date: Sat, 04 Apr 2026 07:16:28 GMT
• Title: The Format Tax
• Abstract summary: We find that structured output requirements substantially degrade reasoning and writing performance across open-weight models.<n>This diagnosis points to a simple principle: decouple reasoning from formats.<n>Most recent closed-weight models show little to no format tax, suggesting the problem is not inherent to structured generation but a gap that current open-weight models have yet to close.
• Score: 30.128781228264216
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Asking a large language model to respond in JSON should be a formatting choice, not a capability tax. Yet we find that structured output requirements -- JSON, XML, LaTeX, Markdown -- substantially degrade reasoning and writing performance across open-weight models. The research response has focused on constrained decoding, but sampling bias accounts for only a fraction of the degradation. The dominant cost enters at the prompt: format-requesting instructions alone cause most of the accuracy loss, before any decoder constraint is applied. This diagnosis points to a simple principle: decouple reasoning from formatting. Whether by generating freeform first and reformatting in a second pass, or by enabling extended thinking within a single generation, separating the two concerns substantially recovers lost accuracy. Across six open-weight models, four API models, four formats, and tasks spanning math, science, logic, and writing, decoupling recovers most lost accuracy. Notably, most recent closed-weight models show little to no format tax, suggesting the problem is not inherent to structured generation but a gap that current open-weight models have yet to close. Code is available at https://github.com/ivnle/the-format-tax.

Related papers

• Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models [0.0]
  Two experiments cross all three factors on one held, contamination-free synthetic corpus.<n>Experiment 1 measures instruction-following decay as rule count N grows from 10 to 160, crossed with four formats and system-prompt vs. user-turn placement.<n>Experiment 2 measures recall accuracy, false-premise sycophancy, and absent-fact fabrication across a 2k-to-512k-token context ladder.
  arXiv  Detail & Related papers  (2026-07-21T16:31:35Z)
• CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression [77.41779716950387]
  We present Cavewoman, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and reference-text agreement.<n>We evaluate eight models on five datasets at five reduction levels, with both channels measured on the same items.
  arXiv  Detail & Related papers  (2026-06-23T02:56:48Z)
• Capacity, Not Format: Rethinking Structured Reasoning Failures [0.0]
  Prior work treats structured output as a reasoning tax, but this framing is incomplete.<n>Using information-matched prose controls and a four-level schema gradient, we separate format-specific effects from prompt-length confounds.<n>We find that structured formats are capacity-dependent.
  arXiv  Detail & Related papers  (2026-06-08T12:26:54Z)
• Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation [0.0]
  Large language models pick up social biases from the data they are trained on and carry those biases into downstream applications.<n>In this paper we take a different route: we debias the model at decoding time, treating bias mitigation as a structured search over candidate tokens without ever touching model weights.<n>We design three schemes of increasing sophistication (Best-of-N selection, Sequential critique-and-revise, and Constitutional self-audit) and evaluate them on four models.
  arXiv  Detail & Related papers  (2026-05-04T08:51:34Z)
• Token-Oriented Object Notation vs JSON: A Benchmark of Plain and Constrained Decoding Generation [0.0]
  Token-Oriented Object Notation (TOON) aims to replace as a serialization format for passing structured data to LLMs with significantly reduced token usage.<n>To test this, we conducted a benchmark creating several test cases with regard to structural complexity, a validation and comparing plain generation vs structured output.
  arXiv  Detail & Related papers  (2026-02-08T11:58:03Z)
• Breaking the Myth: Can Small Models Infer Postconditions Too? [15.725275719200303]
  We show that a small, fine-tuned language model can achieve high-quality postcondition generation with much lower computational costs.<n>Our approach tackles real-world repository dependencies and preserves pre-state information, allowing for expressive and accurate specifications.
  arXiv  Detail & Related papers  (2025-07-14T11:44:04Z)
• Format-Adapter: Improving Reasoning Capability of LLMs by Adapting Suitable Format [93.99600697438079]
  Prior works have shown that multiple reasoning formats outperform a single format when generating multiple answers.<n>We adapt suitable formats to the given tasks by generating and selecting formats.<n>We conduct experiments on math and commonsense reasoning tasks, where Format-Adapter achieves a 4.3% performance improvement on average over previous works.
  arXiv  Detail & Related papers  (2025-06-29T08:11:52Z)
• Studying and Recommending Information Highlighting in Stack Overflow Answers [47.98908661334215]
  We studied 31,169,429 answers of Stack Overflow. For training recommendation models, we choose CNN-based and BERT-based models for each type of formatting. Our models achieve a precision ranging from 0.50 to 0.72 for different formats.
  arXiv  Detail & Related papers  (2024-01-03T00:13:52Z)
• Gradient-Free Structured Pruning with Unlabeled Data [57.999191898036706]
  We propose a gradient-free structured pruning framework that uses only unlabeled data. Up to 40% of the original FLOP count can be reduced with less than a 4% accuracy loss across all tasks considered.
  arXiv  Detail & Related papers  (2023-03-07T19:12:31Z)
• Speculative Decoding with Big Little Decoder [108.95187338417541]
  Big Little Decoder (BiLD) is a framework that can improve inference efficiency and latency for a wide range of text generation applications. On an NVIDIA T4 GPU, our framework achieves a speedup of up to 2.12x speedup with minimal generation quality degradation. Our framework is fully plug-and-play and can be applied without any modifications in the training process or model architecture.
  arXiv  Detail & Related papers  (2023-02-15T18:55:29Z)
• A Conditional Cascade Model for Relational Triple Extraction [0.9926500244448218]
  Tagging based methods are one of the mainstream methods in triple extraction. Most of them suffer from the class imbalance issue greatly. We propose a novel tagging based model that addresses this issue.
  arXiv  Detail & Related papers  (2021-08-20T03:00:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.