PhantomFill: When the Form Demands an Answer, Language Models Invent One

• URL: http://arxiv.org/abs/2607.20492v1
• Date: Thu, 11 Jun 2026 12:15:05 GMT
• Title: PhantomFill: When the Form Demands an Answer, Language Models Invent One
• Abstract summary: Language models in production do not write prose.<n>We ask thirteen models the same question and change only the answer format.<n>We show that the form itself causes hallucination.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language models in production do not write prose. They fill forms: JSON fields, function arguments, extraction templates. We show that the form itself causes hallucination. We ask thirteen models the same question about the same input and change only the answer format. The inputs are built so the question cannot be answered: a viral post showing 12,400 likes but no visible replies, a support ticket whose call was never transcribed. In free text, GPT-5.5 answers honestly. It says there is no reply data, 98% of the time. Given a required JSON field for sentiment, the same model invents an answer 40 times out of 40. It fabricates the mood of crowds it never saw and quotes customers it never heard. The pattern holds with force. Required fields drive fabrication to 100% in ten of thirteen models. An explicit "insufficient evidence" option rescues only the frontier: all nine open-weight models ignore it. A direct instruction, do not infer sentiment, is overridden by the schema in four of six models. Resistance does not come with scale: within a single model family, the smallest model refuses, the mid-sized model fabricates, the largest refuses again. Honesty under format pressure is a training outcome that no one is measuring. The fabrication hides exactly where hedging is impossible: in required enums and minimum-count arrays, fields where no disclaimer fits. We release PhantomFill, a benchmark with deterministic scoring and two reportable numbers: the Coerced Fabrication Rate and the Escape Utilization Rate. The fix we test is one line of schema. The failure we measure is everywhere.

Related papers

• Prompt-Model Interaction Reaches the Fixed Points: A deterministic, task-free structural readout -- and the factorizations of it that failed [0.0]
  We ask a readout with no task in it: the fixed-point structure of the short-window argmax map x_t+1 = argmax_x p(x | x_t-1, x_t), censused from 96 starts.<n>It is deterministic, so nothing can be helped or hurt, and it exists only at short windows -- four of six models lose it entirely by window 16 -- so everything here concerns how a model reads a fragment.<n>First, the interaction reaches this readout at full magnitude: nine tokens of conditioning move the fixed-point fraction across most of its
  arXiv  Detail & Related papers  (2026-08-21T17:25:18Z)
• Why LLMs Give In: Conversational Factors and Reasoning Behind Medical Sycophancy [8.386215461053723]
  A language model that abandons a correct medical answer under user pushback is more dangerous than one that was simply wrong.<n>We study medical sycophancy in language models with a fully crossed factorial design over four conversational factors.
  arXiv  Detail & Related papers  (2026-08-02T05:49:25Z)
• Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models [0.0]
  Two experiments cross all three factors on one held, contamination-free synthetic corpus.<n>Experiment 1 measures instruction-following decay as rule count N grows from 10 to 160, crossed with four formats and system-prompt vs. user-turn placement.<n>Experiment 2 measures recall accuracy, false-premise sycophancy, and absent-fact fabrication across a 2k-to-512k-token context ladder.
  arXiv  Detail & Related papers  (2026-07-21T16:31:35Z)
• Structured Output Collapses Answer Diversity Across 44 Language Models [3.7734715043126488]
  When a language model must choose one answer from a large space of equally valid options, a format clause -- "Reply with only" -- changes which answer it chooses.<n>We re-run the One-Word Census (arXiv:2607.97): 31 wide-answer-space category prompts asked of 44 models, now with the reply requested in.<n> Convergence deepens sharply: on the unconstrained "Pick a word" prompt the modal answer rises from 41% to 64% of the pool and distinct answers fall from 52 to 36.
  arXiv  Detail & Related papers  (2026-07-20T19:47:16Z)
• The One-Word Census: Answer-Choice Conformity Across 44 Language Models [3.7734715043126488]
  We characterize convergence of 31 single-turn prompts, each naming a category with many valid one-word answers.<n>We score each model by answer-choice surprisal: the average $-log2$ probability of its answers under the pooled answers of all other models.<n> Persona- and community-tuned models are the most divergent; the newest mainline flagships are the most conformist.
  arXiv  Detail & Related papers  (2026-07-14T14:12:05Z)
• No Single Best Model for Diversity: Learning a Router for Sample Diversity [69.53166985556759]
  In this paper, we study methods to elicit a comprehensive set of valid responses.<n>We introduce textbfdiversity coverage, a metric that measures the total quality scores assigned to each textbfunique answer.<n>Per each prompt, there exists a model that outperforms all other models significantly at generating a diverse answer set.
  arXiv  Detail & Related papers  (2026-04-02T17:58:37Z)
• Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding [97.05584099530226]
  We introduce MF$2$, a new benchmark for evaluating whether models can comprehend, consolidate, and recall key narrative information from full-length movies.<n>For each pair, models must correctly identify both the true and false claims.<n>Our experiments demonstrate that both open-weight and closed state-of-the-art models fall well short of human performance.
  arXiv  Detail & Related papers  (2025-06-06T17:58:36Z)
• Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models [68.15108215197279]
  A key component of building safe and reliable language models is enabling the models to appropriately refuse to answer certain questions.<n>We propose refusal tokens, one such token for each refusal category or a single refusal token, which are prepended to the model's responses during training.
  arXiv  Detail & Related papers  (2024-12-09T18:40:44Z)
• Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablation [29.605302471407537]
  Training a language model to be both helpful and harmless requires careful calibration of refusal behaviours.<n>We propose a simple and surgical method for mitigating false refusal in language models via single vector ablation.<n>Our approach is training-free and model-agnostic, making it useful for mitigating the problem of false refusal in current and future language models.
  arXiv  Detail & Related papers  (2024-10-04T13:25:32Z)
• Discovering Latent Knowledge in Language Models Without Supervision [72.95136739040676]
  Existing techniques for training language models can be misaligned with the truth. We propose directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way. We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models.
  arXiv  Detail & Related papers  (2022-12-07T18:17:56Z)
• TruthfulQA: Measuring How Models Mimic Human Falsehoods [2.7143159361691227]
  We propose a benchmark to measure whether a language model is truthful in generating answers to questions. The benchmark comprises 817 questions that span 38 categories, including health, law, finance and politics. The best model was truthful on 58% of questions, while human performance was 94%.
  arXiv  Detail & Related papers  (2021-09-08T17:15:27Z)
• Meaningful Answer Generation of E-Commerce Question-Answering [77.89755281215079]
  In e-commerce portals, generating answers for product-related questions has become a crucial task. In this paper, we propose a novel generative neural model, called the Meaningful Product Answer Generator (MPAG) MPAG alleviates the safe answer problem by taking product reviews, product attributes, and a prototype answer into consideration.
  arXiv  Detail & Related papers  (2020-11-14T14:05:30Z)
• Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning [50.114651561111245]
  Models often exploit dataset artifacts to produce correct answers, without connecting information across multiple supporting facts. We formalize such undesirable behavior as disconnected reasoning across subsets of supporting facts. Experiments suggest that there hasn't been much progress in multi-hop QA in the reading comprehension setting.
  arXiv  Detail & Related papers  (2020-05-02T11:01:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.