Semantic Content Determines Algorithmic Performance

• URL: http://arxiv.org/abs/2601.21618v1
• Date: Thu, 29 Jan 2026 12:22:17 GMT
• Title: Semantic Content Determines Algorithmic Performance
• Abstract summary: We introduce WhatCounts to test what counts in isolation.<n>WhatCounts is atomic: count items in a list with no duplicates, distractors, or reasoning steps for different semantic types.<n>We show that any LLM function may carry hidden dependencies on the meaning of its inputs.
• Score: 0.19043656221975455
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Counting should not depend on what is being counted; more generally, any algorithm's behavior should be invariant to the semantic content of its arguments. We introduce WhatCounts to test this property in isolation. Unlike prior work that conflates semantic sensitivity with reasoning complexity or prompt variation, WhatCounts is atomic: count items in an unambiguous, delimited list with no duplicates, distractors, or reasoning steps for different semantic types. Frontier LLMs show over 40% accuracy variation depending solely on what is being counted - cities versus chemicals, names versus symbols. Controlled ablations rule out confounds. The gap is semantic, and it shifts unpredictably with small amounts of unrelated fine-tuning. LLMs do not implement algorithms; they approximate them, and the approximation is argument-dependent. As we show with an agentic example, this has implications beyond counting: any LLM function may carry hidden dependencies on the meaning of its inputs.

Related papers

• Underspecification does not imply Incoherence: The Risks of Semantic Collapse in Coding Models [6.72873429875663]
  Large Language Models (LLMs) have become increasingly effective at generating code when task descriptions are clear and precise.<n>Yet, user-provided task descriptions are often ambiguous, incomplete, or contradictory, leaving critical aspects of the intended program behavior underspecified.<n>We find that LLMs frequently collapse onto a single incorrect interpretation of the task description, consistently generating coherent but behaviorally misaligned code.
  arXiv  Detail & Related papers  (2026-07-02T09:43:15Z)
• Towards Human-Like Interactive Speech Recognition With Agentic Correction and Semantic Evaluation [53.844308305341166]
  We propose textbfAgentic ASR, a closed-loop framework that combines a single-pass ASR front-end with semantic correction, intent routing, and reasoning-based editing.<n> Experiments on multilingual, named-entity-intensive, and code-switching benchmarks show that iterative interaction consistently reduces semantic errors.
  arXiv  Detail & Related papers  (2026-05-28T06:23:31Z)
• Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification [56.51953062869371]
  DoVerifier is a symbolic verifier that checks whether causal expressions are derivable from a given causal graph using rules from do-calculus and probability theory.<n>Our evaluations on synthetic data and causal QA benchmarks show that DoVerifier more accurately captures semantic correctness of causal reasoning traces.
  arXiv  Detail & Related papers  (2026-01-29T03:22:58Z)
• Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs [2.2574632480801484]
  Tokenization is the first step in training any Large Language Model (LLM)<n>We propose a set of penalty functions that compute a tokenization penalty for a given text for a specific LLM, indicating how "bad" the tokenization is.
  arXiv  Detail & Related papers  (2025-12-26T09:16:33Z)
• Verbalized Algorithms [0.9697877942346907]
  Instead of querying LLMs in a one-shot manner, we propose a paradigm we call emphverbalized algorithms (VAs)<n>VAs decompose a task into simple elementary operations on natural language strings that they should be able to answer reliably.<n>We demonstrate the effectiveness of this approach on sorting and clustering tasks.
  arXiv  Detail & Related papers  (2025-09-09T21:14:44Z)
• Measuring Scalar Constructs in Social Science with LLMs [48.92998035333579]
  We evaluate approaches to measuring scalar constructs in large language models.<n>We find that pairwise comparisons produce better measurements than simply prompting the LLM to directly output the scores.<n>Finetuning smaller models with as few as 1,000 training pairs can match or exceed the performance of prompted LLMs.
  arXiv  Detail & Related papers  (2025-09-03T08:19:13Z)
• EquivPruner: Boosting Efficiency and Quality in LLM-Based Search via Action Pruning [40.506243479030104]
  EquivPruner identifies and prunes semantically equivalent actions during reasoning search.<n>MathEquiv, the first dataset created for mathematical statement equivalence, enables the training of a lightweight equivalence detector.
  arXiv  Detail & Related papers  (2025-05-22T07:07:43Z)
• QUDsim: Quantifying Discourse Similarities in LLM-Generated Text [70.22275200293964]
  We introduce an abstraction based on linguistic theories in Questions Under Discussion (QUD) and question semantics to help quantify differences in discourse progression.<n>We then use this framework to build $textbfQUDsim$, a similarity metric that can detect discursive parallels between documents.<n>Using QUDsim, we find that LLMs often reuse discourse structures (more so than humans) across samples, even when content differs.
  arXiv  Detail & Related papers  (2025-04-12T23:46:09Z)
• Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models [88.07940818022468]
  We take an initial step on measuring the role of shared semantics among subwords in the encoder-only multilingual language models (mLMs) We form "semantic tokens" by merging the semantically similar subwords and their embeddings. inspections on the grouped subwords show that they exhibit a wide range of semantic similarities.
  arXiv  Detail & Related papers  (2024-11-07T08:38:32Z)
• Can We Count on LLMs? The Fixed-Effect Fallacy and Claims of GPT-4 Capabilities [8.1022073999821]
  We present measurements of GPT-4 performance on several deterministic tasks. We find that seemingly trivial modifications in the task-prompt or input population can yield differences far larger than can be explained by sampling effects.
  arXiv  Detail & Related papers  (2024-09-11T21:48:33Z)
• Contrastive Instruction Tuning [61.97704869248903]
  We propose Contrastive Instruction Tuning to maximize the similarity between semantically equivalent instruction-instance pairs. Experiments on the PromptBench benchmark show that CoIN consistently improves LLMs' robustness to unseen instructions with variations across character, word, sentence, and semantic levels by an average of +2.5% in accuracy.
  arXiv  Detail & Related papers  (2024-02-17T00:09:32Z)
• Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting [87.30837365008931]
  Large language models (LLMs) equipped with Chain-of-Thought (CoT) prompting are able to make accurate incremental predictions even on unscalable tasks. This study examines the impact of LLMs' step-by-step predictions on gender bias in unscalable tasks.
  arXiv  Detail & Related papers  (2024-01-28T06:50:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.