Improving Cross-Format Robustness in Language Models with Multi-Format Training

• URL: http://arxiv.org/abs/2606.11643v1
• Date: Wed, 10 Jun 2026 04:07:41 GMT
• Title: Improving Cross-Format Robustness in Language Models with Multi-Format Training
• Abstract summary: Cross-format robustness is the extent to which a model answers the same underlying question consistently across formats.<n>We compare full-format training with FormatMix, which expands only a subset of training items into multiple equivalent formats.<n>We find that expanding only about 30% of the training set into multiple formats often recovers most of the gain from full-format training.
• Score: 17.87388405562304
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models often remain sensitive to answer format: a question solved correctly in one form may fail in another semantically equivalent form. To study this gap, we define cross-format robustness as the extent to which a model answers the same underlying question consistently across formats. We then compare full-format training with FormatMix, which expands only a subset of training items into multiple equivalent formats using either random or targeted selection. Across GLM4 and Llama-3.1, multi-format supervision consistently improves both task performance and cross-format robustness, whereas Multiple-choice question (MCQ)-only supervision alone brings little benefit and can even reduce robustness. We further find that expanding only about 30% of the training set into multiple formats often recovers most of the gain from full-format training, and this effect appears across the model families and sizes we study. These results suggest that format diversity, rather than additional supervision alone, is the key driver of robustness. That lightweight multi-format augmentation is a practical way to make LLMs less sensitive to answer format without changing the base model.

Related papers

• To Diff or Not to Diff? Structure-Aware and Adaptive Output Formats for Efficient LLM-based Code Editing [54.399335188667045]
  BlockDiff and FuncDiff are structure-aware diff formats that represent changes as block-level rewrites of syntactically coherent units.<n>AdaEdit is a general adaptive edit strategy that trains LLMs to dynamically choose the most token-efficient format between a given diff format and full code.
  arXiv  Detail & Related papers  (2026-04-30T01:14:13Z)
• MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inference [1.938624051379935]
  Quantization-aware training (QAT) is typically performed for a single target numeric format.<n>We study multi-format QAT, where a single model is trained to be robust across multiple quantization formats.<n>We propose the Slice-and-Scale conversion procedure for both MXINT and MXFP that converts a high-precision representation into lower-precision formats without re-training.
  arXiv  Detail & Related papers  (2026-04-01T06:12:19Z)
• Format-Adapter: Improving Reasoning Capability of LLMs by Adapting Suitable Format [93.99600697438079]
  Prior works have shown that multiple reasoning formats outperform a single format when generating multiple answers.<n>We adapt suitable formats to the given tasks by generating and selecting formats.<n>We conduct experiments on math and commonsense reasoning tasks, where Format-Adapter achieves a 4.3% performance improvement on average over previous works.
  arXiv  Detail & Related papers  (2025-06-29T08:11:52Z)
• ReFF: Reinforcing Format Faithfulness in Language Models across Varied Tasks [32.021938679807555]
  We present FormatBench, a format-related benchmark for large language models (LLMs)<n>Experiments on the benchmark reveal that state-of-the-art open- and closed-source LLMs still suffer from severe deficiency in format faithfulness.<n>We propose to Reinforce Format Faithfulness (ReFF) to help LLMs generate formatted output as instructed without compromising general quality.
  arXiv  Detail & Related papers  (2024-12-12T11:03:25Z)
• LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs [69.40865293066885]
  We present the first systematic evaluation examining format bias in performance of large language models (LLMs)<n>We present our empirical format bias evaluation spanning four commonly used categories -- multiple-choice question-answer, wrapping, list, and mapping.
  arXiv  Detail & Related papers  (2024-08-16T10:45:45Z)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting [68.19544657508509]
  Large language models (LLMs) are adopted as a fundamental component of language technologies. We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt format in few-shot settings. We propose an algorithm that rapidly evaluates a sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights.
  arXiv  Detail & Related papers  (2023-10-17T15:03:30Z)
• MatFormer: Nested Transformer for Elastic Inference [91.45687988953435]
  MatFormer is a novel Transformer architecture designed to provide elastic inference across diverse deployment constraints.<n>MatFormer achieves this by incorporating a nested Feed Forward Network (FFN) block structure within a standard Transformer model.<n>We show that a 850M decoder-only MatFormer language model (MatLM) allows us to extract multiple smaller models spanning from 582M to 850M parameters.
  arXiv  Detail & Related papers  (2023-10-11T17:57:14Z)
• LPT: Long-tailed Prompt Tuning for Image Classification [178.52948452353834]
  We introduce several trainable prompts into a frozen pretrained model to adapt it to long-tailed data. In phase 1, we train the shared prompt via supervised prompt tuning to adapt a pretrained model to the desired long-tailed domain. In phase 2, we use the learnt shared prompt as query to select a small best matched set for a group of similar samples.
  arXiv  Detail & Related papers  (2022-10-03T15:47:02Z)
• Transforming Sequence Tagging Into A Seq2Seq Task [10.130389627403433]
  We study different formats one could use for casting input text sentences into the input and target of a Seq2Seq model. We introduce a new format, which we show to not only be simpler but also more effective. We find that the new format is more robust and almost completely devoid of hallucination.
  arXiv  Detail & Related papers  (2022-03-16T03:48:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.