The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval

• URL: http://arxiv.org/abs/2605.07186v1
• Date: Fri, 08 May 2026 03:26:42 GMT
• Title: The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval
• Abstract summary: We study how word-boundary corruption affects how Large Language Model benchmarks detect targeted information.<n>By inserting whitespace characters within words to break them into fragments, LLMs' detection accuracy follows a U-shaped curve with the increase in insertion rate.
• Score: 21.243421703047037
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing Large Language Model (LLM) benchmarks primarily focus on syntactically correct inputs, leaving a significant gap in evaluation on imperfect text. In this work, we study how word-boundary corruption affects how LLMs detect targeted information. By inserting whitespace characters within words to break them into fragments, LLMs' detection accuracy follows a U-shaped curve with the increase in insertion rate. We refer to this curve as the Text Uncanny Valley. To explain such observation, we propose a mode transition hypothesis: LLMs operate in a word-level mode for near-normal text and a character-level mode for heavily fragmented text, with the valley marking the disordered transition where neither mode is effective. Four experiments and one analysis are consistent with this account: in-context learning fails to rescue valley-bottom performance; regularizing the perturbation substantially reduces the U-shape; a math reasoning task replicates the U-shape for Gemini 3.0 Flash but not for stronger models, suggesting the effect is attenuated when tasks rely less on exact lexical alignment; and tokenization entropy peaks before the F1 minimum, consistent with a regime-conflict interpretation. These findings reveal a failure mode invisible to clean-text benchmarks yet directly relevant to any deployment scenario involving noisy or uncurated text inputs.

Related papers

• Data and Evaluation Closed-Loop for Model Capability Enhancement [8.245706308161504]
  We show that evaluation can be routine, auditable, and experimentally validated rather than intuitive.<n>We show that the evaluation-to-data inference can be routine, auditable, and experimentally validated rather than intuitive.
  arXiv  Detail & Related papers  (2026-06-26T14:45:57Z)
• Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models [14.752088383510788]
  Vision-language models (VLMs) are often deployed on text-only inputs, although they are trained with images.<n>We find that removing the vision modality causes large drops in accuracy and severe miscalibration, and the model does not behave like its original language backbone under text-only prompting.<n>We propose the Latent Imagination Module (LIM), a lightweight cross-attention module that predicts imagined latent embeddings from textual input and feeds them into a frozen VLM backbone without pixel-level image synthesis.
  arXiv  Detail & Related papers  (2026-04-03T10:03:02Z)
• From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks [49.57538588967748]
  Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks.<n>We uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma.<n>We show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph.
  arXiv  Detail & Related papers  (2026-04-01T11:40:12Z)
• PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise [60.63315470285562]
  MiniTruePrefixes is a novel specialized model that better detects factual inconsistencies over text prefixes.<n>We show that integrating MiniTruePrefixes into a controlled decoding framework substantially improves factual consistency in abstractive summarization.
  arXiv  Detail & Related papers  (2025-11-03T09:07:44Z)
• RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns [50.401907401444404]
  Large language models (LLMs) are crucial for preventing misuse and building trustworthy AI systems.<n>We propose RepreGuard, an efficient statistics-based detection method.<n> Experimental results show that RepreGuard outperforms all baselines with average 94.92% AUROC on both in-distribution (ID) and OOD scenarios.
  arXiv  Detail & Related papers  (2025-08-18T17:59:15Z)
• Extract Free Dense Misalignment from CLIP [7.0247398611254175]
  This work proposes a novel approach, dubbed CLIP4DM, for detecting dense misalignments from pre-trained CLIP.<n>We revamp the gradient-based attribution computation method, enabling negative gradient of individual text tokens to indicate misalignment.<n>Our method demonstrates state-of-the-art performance among zero-shot models and competitive performance with fine-tuned models.
  arXiv  Detail & Related papers  (2024-12-24T12:51:05Z)
• Vulnerability of LLMs to Vertically Aligned Text Manipulations [130.54118945532898]
  Vertical text input is commonly encountered in various real-world applications, such as mathematical computations and word-based Sudoku puzzles.<n>Recent research demonstrates that modifying input formats, such as vertically aligning words for encoder-based models, can substantially lower accuracy in text classification tasks.
  arXiv  Detail & Related papers  (2024-10-26T00:16:08Z)
• Robustness of Large Language Models to Perturbations in Text [2.2734015467359217]
  Large language models (LLMs) have shown impressive performance, but can they handle the inevitable noise in real-world data?<n>This work tackles this critical question by investigating LLMs' resilience against morphological variations in text.<n>Our findings show that contrary to popular beliefs, generative LLMs are quiet robust to noisy perturbations in text.
  arXiv  Detail & Related papers  (2024-07-12T04:50:17Z)
• A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection [63.56136319976554]
  Large Language Models (LLMs) generate hallucinations, which can cause significant damage when deployed for mission-critical tasks. We propose a self-check approach based on reverse validation to detect factual errors automatically in a zero-resource fashion. We empirically evaluate our method and existing zero-resource detection methods on two datasets.
  arXiv  Detail & Related papers  (2023-10-10T10:14:59Z)
• ContourNet: Taking a Further Step toward Accurate Arbitrary-shaped Scene Text Detection [147.10751375922035]
  We propose the ContourNet, which effectively handles false positives and large scale variance of scene texts. Our method effectively suppresses these false positives by only outputting predictions with high response value in both directions.
  arXiv  Detail & Related papers  (2020-04-10T08:15:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.