Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models

• URL: http://arxiv.org/abs/2511.21086v1
• Date: Wed, 26 Nov 2025 06:12:33 GMT
• Title: Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models
• Abstract summary: Large language models must satisfy hard orthographic constraints during controlled text generation.<n>We evaluate 28 configurations spanning three model families on 58 word puzzles requiring character-level constraint satisfaction.
• Score: 2.3991974633684854
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models must satisfy hard orthographic constraints during controlled text generation, yet systematic cross-architecture evaluation remains limited. We evaluate 28 configurations spanning three model families (Qwen3, Claude Haiku-4.5, GPT-5-mini) on 58 word puzzles requiring character-level constraint satisfaction. Architectural differences produce substantially larger performance gaps (2.0-2.2x, F1=0.761 vs. 0.343) than parameter scaling within families (83% gain from eightfold scaling), suggesting that constraint satisfaction may require specialized architectural features or training objectives beyond standard language model scaling. Thinking budget sensitivity proves heterogeneous: high-capacity models show strong returns (+0.102 to +0.136 F1), while mid-sized variants saturate or degrade. These patterns are inconsistent with uniform compute benefits. Using difficulty ratings from 10,000 human solvers per puzzle, we establish modest but consistent calibration (r=0.24-0.38) across all families, yet identify systematic failures on common words with unusual orthography ("data", "poop", "loll": 86-95% human success, 89-96% model miss rate). These failures reveal over-reliance on distributional plausibility that penalizes orthographically atypical but constraint-valid patterns, suggesting architectural innovations may be required beyond simply scaling parameters or computational budgets.

Related papers

• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym [13.424353009036068]
  Spatial-Gym tests pathfinding in 2D-grid puzzles as a sequential decision task with optional backtracking.<n>We evaluate eight models in three settings (one-shot, step-by-step, step-by-step with backtracking) against human, random, and A* baselines on 500 episodes.
  arXiv  Detail & Related papers  (2026-04-10T14:05:50Z)
• Severity-Aware Weighted Loss for Arabic Medical Text Generation [0.25489046505746704]
  We propose a severity-aware weighted loss for fine-tuning Arabic language models on medical complaint-response data.<n>Experiments are conducted using the MAQA dataset, which provides Arabic medical complaints and trusted human responses.
  arXiv  Detail & Related papers  (2026-04-07T18:26:41Z)
• Did You Forget What I Asked? Prospective Memory Failures in Large Language Models [0.0]
  Large language models often fail to satisfy formatting instructions when they must simultaneously perform demanding tasks.<n>We study this behaviour through a prospective memory inspired lens from cognitive psychology, using a controlled paradigm.<n>Vulnerability is highly type-dependent, with drops up to 50%, while avoidance constraints remain comparatively robust.<n>A salience-enhanced format (explicit instruction framing plus a trailing reminder) recovers much of the lost compliance, restoring performance to 90-100% in many settings.
  arXiv  Detail & Related papers  (2026-03-07T05:58:19Z)
• From Abstract to Contextual: What LLMs Still Cannot Do in Mathematics [79.81905350372067]
  We study gap through contextual mathematical reasoning.<n>We introduce ContextMATH, a benchmark that repurposes AIME and MATH-500 problems into two contextual settings.<n>Open-source models decline by 13 and 34 points on SG and CS, while proprietary models drop by 13 and 20.
  arXiv  Detail & Related papers  (2026-01-30T14:56:04Z)
• Silhouette-based Gait Foundation Model [56.27974816297294]
  Building a unified gait foundation model requires addressing two longstanding barriers: Scalability and Generalization.<n>We introduce FoundationGait, the first scalable, self-supervised pretraining framework for gait understanding.
  arXiv  Detail & Related papers  (2025-11-30T01:53:41Z)
• Capability Ceilings in Autoregressive Language Models: Empirical Evidence from Knowledge-Intensive Tasks [0.2538209532048866]
  We document capability ceilings in decoder-only autoregressive language models across knowledge-intensive tasks.<n>We quantify capability-specific scaling failures in OPT and Pythia model families to inform resource allocation decisions.
  arXiv  Detail & Related papers  (2025-10-23T11:09:31Z)
• Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures [87.75098311090642]
  Current preference learning methods achieve high accuracy on standard benchmarks but exhibit significant performance degradation when objective quality signals are removed.<n>We introduce WritingPreferenceBench, a dataset of 1,800 human-annotated preference pairs (1,200 English, 600 Chinese) across 8 creative writing genres.
  arXiv  Detail & Related papers  (2025-10-16T12:23:13Z)
• BeyondBench: Benchmark-Free Evaluation of Reasoning in Language Models [13.380359214677176]
  We introduce BeyondBench, an evaluation framework that avoids contamination from internet-scale training data.<n>Our framework covers 44 algorithmic tasks with a total of 117 variations, grouped into three difficulty levels.<n>We evaluate 101 language models, including 85 open-source and 16 closed-source models, spanning sizes from 0.5B to 141B parameters.
  arXiv  Detail & Related papers  (2025-09-29T02:49:01Z)
• Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT? [59.418994222096885]
  We conduct a detailed analysis of model performance on the AIME24 dataset.<n>We categorize questions into four tiers (Easy, Medium, Hard, and Extremely Hard)<n>We find that progression from Easy to Medium tier requires adopting an R1 reasoning style with minimal SFT-1K instances.<n>Exh-level questions present a fundamentally different challenge; they require unconventional problem-solving skills.
  arXiv  Detail & Related papers  (2025-04-16T03:39:38Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Hybrid-Segmentor: A Hybrid Approach to Automated Fine-Grained Crack Segmentation in Civil Infrastructure [52.2025114590481]
  We introduce Hybrid-Segmentor, an encoder-decoder based approach that is capable of extracting both fine-grained local and global crack features. This allows the model to improve its generalization capabilities in distinguish various type of shapes, surfaces and sizes of cracks. The proposed model outperforms existing benchmark models across 5 quantitative metrics (accuracy 0.971, precision 0.804, recall 0.744, F1-score 0.770, and IoU score 0.630), achieving state-of-the-art status.
  arXiv  Detail & Related papers  (2024-09-04T16:47:16Z)
• Integrating Large Language Models into a Tri-Modal Architecture for Automated Depression Classification on the DAIC-WOZ [0.0]
  Major Depressive Disorder (MDD) is a pervasive mental health condition that affects 300 million people worldwide. This work presents a novel, BiLSTM-based tri-modal model-level fusion architecture for the binary classification of depression from clinical interview recordings.
  arXiv  Detail & Related papers  (2024-07-27T21:00:36Z)
• Uncovering Weaknesses in Neural Code Generation [21.552898575210534]
  We assess the quality of generated code using match-based and execution-based metrics, then conduct thematic analysis to develop a taxonomy of nine types of weaknesses. In the CoNaLa dataset, inaccurate prompts are a notable problem, causing all large models to fail in 26.84% of cases. Missing pivotal semantics is a pervasive issue across benchmarks, with one or more large models omitting key semantics in 65.78% of CoNaLa tasks. All models struggle with proper API usage, a challenge amplified by vague or complex prompts.
  arXiv  Detail & Related papers  (2024-07-13T07:31:43Z)
• Identifying and Mitigating Model Failures through Few-shot CLIP-aided Diffusion Generation [65.268245109828]
  We propose an end-to-end framework to generate text descriptions of failure modes associated with spurious correlations. These descriptions can be used to generate synthetic data using generative models, such as diffusion models. Our experiments have shown remarkable textbfimprovements in accuracy ($sim textbf21%$) on hard sub-populations.
  arXiv  Detail & Related papers  (2023-12-09T04:43:49Z)
• Switchable Representation Learning Framework with Self-compatibility [50.48336074436792]
  We propose a Switchable representation learning Framework with Self-Compatibility (SFSC) SFSC generates a series of compatible sub-models with different capacities through one training process. SFSC achieves state-of-the-art performance on the evaluated datasets.
  arXiv  Detail & Related papers  (2022-06-16T16:46:32Z)
• (Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performance [7.2223831790495625]
  We propose to re-evaluate morphological inflection models by employing harder train-test splits. Experiments with the three top-ranked systems on the SIGMORPHON's 2020 shared-task show that the lemma-split presents an average drop of 30 percentage points.
  arXiv  Detail & Related papers  (2021-08-12T12:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.