How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction

• URL: http://arxiv.org/abs/2601.08626v2
• Date: Sat, 17 Jan 2026 09:18:48 GMT
• Title: How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstruction
• Abstract summary: Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored.<n>We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean.<n>We propose a diagnostic framework that evaluates models beyond recovery accuracy, including semantic fidelity, logical validity, robustness, sensitivity, and information density.
• Score: 26.53848099802812
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) excel at semantic understanding, yet their ability to reconstruct internal structure from scrambled inputs remains underexplored. Sentence-level restoration is ill-posed for automated evaluation because multiple valid word orders often exist. We introduce OrderProbe, a deterministic benchmark for structural reconstruction using fixed four-character expressions in Chinese, Japanese, and Korean, which have a unique canonical order and thus support exact-match scoring. We further propose a diagnostic framework that evaluates models beyond recovery accuracy, including semantic fidelity, logical validity, consistency, robustness sensitivity, and information density. Experiments on twelve widely used LLMs show that structural reconstruction remains difficult even for frontier systems: zero-shot recovery frequently falls below 35%. We also observe a consistent dissociation between semantic recall and structural planning, suggesting that structural robustness is not an automatic byproduct of semantic competence.

Related papers

• Lost in Reordering: Structural Sensitivity of Multilingual LLMs under Semantics-Preserving Perturbations [19.00420862366793]
  Large Language Models (LLMs) demonstrate strong multilingual reasoning performance, yet their robustness to semantics-preserving structural variation remains underexplored.<n>We investigate the structural sensitivity of multilingual LLMs using two linguistically grounded perturbation settings in Hindi and Malayalam.<n>We observe consistent and significant degradation in mathematical reasoning performance under structurally perturbed inputs.
  arXiv  Detail & Related papers  (2026-09-03T08:10:43Z)
• DHRCL:Training Code LLMs with Dense Hierarchical Rewards and Curriculum Learning [15.421698703970167]
  We propose DHRCL, a reinforcement learning framework with Dense Hierarchical Rewards and Curriculum Learning.<n>DHRCL decomposes feedback into syntax validation, execution success, unit-test pass rate, and AST-based structural similarity.<n>We evaluate DHRCL across Qwen3-4B, Qwen3-8B, and Qwen3-14B backbones, showing that its advantage remains consistent as model capacity increases.
  arXiv  Detail & Related papers  (2026-07-29T04:16:33Z)
• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• Deep Reasoning in General Purpose Agents via Structured Meta-Cognition [58.185853639335896]
  We introduce an inference-time approach for constructing task-specific scaffolds through structured meta-reasoning.<n>We instantiate this approach in a general-purpose agent that distributes complex tasks across more controlled reasoning threads.
  arXiv  Detail & Related papers  (2026-05-12T01:21:37Z)
• Discovering Semantic Latent Structures in Psychological Scales: A Response-Free Pathway to Efficient Simplification [7.405170407676887]
  We introduce a topic-modeling framework that operationalizes semantic latent structure for scale simplification.<n>Items are encoded using contextual sentence embeddings and grouped via density-based clustering.<n>We benchmarked the framework across DASS, IPIP, and EPOCH, evaluating structural recovery, internal consistency, factor congruence, correlation preservation, and reduction efficiency.
  arXiv  Detail & Related papers  (2026-02-13T03:37:15Z)
• Task-Awareness Improves LLM Generations and Uncertainty [48.857040212979484]
  Bayes-optimal responses consistently outperform standard decoding methods like beam search.<n>Our decision-theoretic framework is applicable to any problem that admits a latent response structure.
  arXiv  Detail & Related papers  (2026-01-29T10:16:23Z)
• Structured Reasoning for Large Language Models [59.215789462977206]
  We propose Structured Reasoning (SCR), a framework that decouples reasoning trajectories into explicit, evaluable, and trainable components.<n>SCR substantially improves reasoning efficiency and self-verification.<n>Compared with existing reasoning paradigms, it reduces output token length by up to 50%.
  arXiv  Detail & Related papers  (2026-01-12T04:04:01Z)
• ReForm: Reflective Autoformalization with Prospective Bounded Sequence Optimization [73.0780809974414]
  We propose a Reflective Autoformalization method that integrates semantic consistency evaluation into the autoformalization process.<n>This enables the model to iteratively generate formal statements, assess its semantic fidelity, and self-correct identified errors.<n>Experiments show that ReForm achieves an average improvement of 22.6 percentage points over the strongest baselines.
  arXiv  Detail & Related papers  (2025-10-28T16:22:54Z)
• Automated Skill Decomposition Meets Expert Ontologies: Bridging the Granularity Gap with LLMs [1.2891210250935148]
  This paper investigates automated skill decomposition using Large Language Models (LLMs)<n>Our framework standardizes the pipeline from prompting and generation to normalization and alignment with ontology nodes.<n>To evaluate outputs, we introduce two metrics: a F1-score that uses optimal embedding-based matching to assess content accuracy, and a hierarchy-aware F1-score that credits structurally correct placements to assess granularity.
  arXiv  Detail & Related papers  (2025-10-13T12:03:06Z)
• Effects of structure on reasoning in instance-level Self-Discover [0.0]
  This paper introduces iSelf-Discover, an instance-level adaptation of the Self-Discover framework, and using it compares dynamically generated structured reasoning with its unstructured counterpart.<n>Our empirical evaluation across diverse benchmarks using state-of-the-art open-source models supports a consistent advantage for unstructured reasoning.
  arXiv  Detail & Related papers  (2025-07-04T07:28:42Z)
• Beyond Surface Similarity: Evaluating LLM-Based Test Refactorings with Structural and Semantic Awareness [15.677544288705883]
  Large Language Models (LLMs) are increasingly used to improve readability and structure while preserving behavior.<n>We propose CTSES, a first step toward human-aligned evaluation of LLMs.<n>CTSES combines CodeBLEU, METEOR, and ROUGE-L into a composite score that balances semantics, lexical clarity, and structural alignment.
  arXiv  Detail & Related papers  (2025-06-07T11:18:17Z)
• How Far Are We from Generating Missing Modalities with Foundation Models? [49.425856207329524]
  We propose an agentic framework tailored for missing modality reconstruction.<n>Our method reduces FID for missing image reconstruction by at least 14% and MER for missing text reconstruction by at least 10% compared to baselines.
  arXiv  Detail & Related papers  (2025-06-04T03:22:44Z)
• Can LLMs Reason Structurally? An Evaluation via the Lens of Data Structures [21.390740746718947]
  We introduce DSR-Bench, the first benchmark to systematically evaluate large language models' structural reasoning.<n>The benchmark spans 20 data structures, 35 operations, and 4,140 synthetically generated problem instances with minimal contamination.
  arXiv  Detail & Related papers  (2025-05-29T23:24:53Z)
• Large Language Models as Quasi-crystals: Coherence Without Repetition in Generative Text [0.0]
  essay proposes an analogy between large language models (LLMs) and quasicrystals, systems that exhibit global coherence without periodic repetition, generated through local constraints.<n> Drawing on the history of quasicrystals, it highlights an alternative mode of coherence in generative language: constraint-based organization without repetition or symbolic intent.<n>This essay aims to reframe the current discussion around large language models, not by rejecting existing methods, but by suggesting an additional axis of interpretation grounded in structure rather than semantics.
  arXiv  Detail & Related papers  (2025-04-16T11:27:47Z)
• StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs [78.84060166851805]
  StructTest is a novel benchmark that evaluates large language models (LLMs) on their ability to follow compositional instructions and generate structured outputs.<n> Assessments are conducted deterministically using a rule-based evaluator, which can be easily extended to new tasks and datasets.<n>We demonstrate that StructTest remains challenging even for top-performing models like Deepseek-V3/R1 and GPT-4o.
  arXiv  Detail & Related papers  (2024-12-23T22:08:40Z)
• Autoregressive Structured Prediction with Language Models [73.11519625765301]
  We describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at.
  arXiv  Detail & Related papers  (2022-10-26T13:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.