Can Perplexity Serve as a Cognitive Signal for Code Understandability?

• URL: http://arxiv.org/abs/2608.00624v1
• Date: Sat, 01 Aug 2026 12:29:42 GMT
• Title: Can Perplexity Serve as a Cognitive Signal for Code Understandability?
• Abstract summary: Recent work suggests that token-level perplexity from large language models can align with localized human confusion during code comprehension.<n>This raises a natural question: can perplexity also serve as a snippet-level signal for code understandability?<n>We conduct an empirical study of this question across multiple human-grounded datasets.
• Score: 13.163098563588727
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work suggests that token-level perplexity from large language models can align with localized human confusion during code comprehension. This raises a natural question: can perplexity also serve as a snippet-level signal for code understandability? We conduct an empirical study of this question across multiple human-grounded datasets, including method-level understandability judgments, output-prediction tasks, and accepted understandability-improvement patches. Despite prior token-level evidence, we find that simple snippet-level aggregations of token perplexity, such as average, median, or peak perplexity, do not reliably correlate with human understandability. We then investigate why this happens. First, token perplexity is highly skewed and heavy-tailed across code structures; extreme spikes arise not only from semantically meaningful constructs, but also from identifiers, literals, types, separators, and tokenization artifacts. Second, human understandability labels often lack consensus, making whole-snippet difficulty a noisy target. Third, perplexity distributions and their alignment with human difficulty vary substantially across models and tokenizers. These findings explain why prior token-level perplexity--confusion alignment does not directly transfer to snippet-level understandability. Overall, our study positions perplexity as a promising but delicate cognitive signal: useful for localized code confusion, but requiring code-aware aggregation, consensus-aware evaluation, and model-sensitivity analysis before it can support reliable code-understandability measurement.

Related papers

• Xetrieval: Mechanistically Explaining Dense Retrieval [36.8058388046157]
  We propose textitXetrieval, an embedding-level mechanistic framework for explaining dense retrieval.<n>textitXetrieval first introduces a lightweight reasoning internalizer that approximates Chain-of-Thought reasoning directly in the embedding space.<n>It then decomposes these reasoning-enhanced embeddings into sparse, human-interpretable features, each associated with a coherent natural language description.
  arXiv  Detail & Related papers  (2026-05-28T07:29:58Z)
• XSearch: Explainable Code Search via Concept-to-Code Alignment [30.606443667402335]
  We propose XSearch, an intrinsically explainable code search framework.<n>XSearch identifies functional concepts in the query and explicitly aligns them with corresponding code statements.<n>We train an encoder with explicit concept-alignment objectives and perform retrieval through explicit matching between query concepts and code statements.
  arXiv  Detail & Related papers  (2026-05-15T15:17:43Z)
• Why Code, Why Now: Learnability, Computability, and the Real Limits of Machine Learning [0.38677478583601776]
  We propose a five-level hierarchy of learnability based on information structure.<n>We argue that the ceiling on ML progress depends less on model size than on whether a task is learnable at all.
  arXiv  Detail & Related papers  (2026-02-15T00:14:31Z)
• Decomposing Query-Key Feature Interactions Using Contrastive Covariances [75.38737409771085]
  We study the query-key space -- the bilinear joint embedding space between queries and keys.<n>It is when features in keys and queries align in these low-rank subspaces that high attention scores are produced.
  arXiv  Detail & Related papers  (2026-02-04T16:50:02Z)
• Beyond Embeddings: Interpretable Feature Extraction for Binary Code Similarity [3.9488518969307305]
  We use a language model-based agent to conduct structured reasoning analysis of assembly code.<n>Unlike hand-crafted features, they are richer and adaptive. Unlike embeddings, they are human-readable, maintainable, and directly searchable with inverted or relational indexes.
  arXiv  Detail & Related papers  (2025-09-27T18:34:32Z)
• LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layers [53.43862310647276]
  Large language models (LLMs) excel at natural language understanding and generation but remain vulnerable to factual errors.<n>We introduce a token-aware, layer-localized contrastive decoding method that aligns specific token types with their most influential transformer layers to improve factual generation.<n>Our method requires no additional training or model modification, and experiments demonstrate that our method consistently improves factuality across multiple LLMs and various benchmarks.
  arXiv  Detail & Related papers  (2025-07-06T14:35:43Z)
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations [83.93566096400723]
  We find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization.<n>Character-level segmentation improves string manipulation and code understanding tasks by up to +14%.<n>Right-aligned digit grouping enhances large-number arithmetic by +33%.
  arXiv  Detail & Related papers  (2025-06-23T18:02:26Z)
• Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space [62.54887038032942]
  We introduce Soft Thinking, a training-free method that emulates human-like "soft" reasoning by generating soft, abstract concept tokens.<n>These concept tokens are created by the probability-weighted mixture of token embeddings, which form the continuous concept space.<n>In essence, each generated concept token encapsulates multiple meanings from related discrete tokens, implicitly exploring various reasoning paths to converge.
  arXiv  Detail & Related papers  (2025-05-21T17:29:15Z)
• Personalized Code Readability Assessment: Are We There Yet? [6.401191523681341]
  Previous work defined approaches to automatically assess code readability that can warn developers when some code artifacts become unreadable.<n>This paper aims to understand to what extent it is possible to assess code readability as subjectively perceived by developers.<n>Our results, however, show that such an approach achieves worse results than a state-of-the-art feature-based model that is trained to work at the snippet-level.
  arXiv  Detail & Related papers  (2025-03-10T21:37:15Z)
• Towards Unsupervised Recognition of Token-level Semantic Differences in Related Documents [61.63208012250885]
  We formulate recognizing semantic differences as a token-level regression task. We study three unsupervised approaches that rely on a masked language model. Our results show that an approach based on word alignment and sentence-level contrastive learning has a robust correlation to gold labels.
  arXiv  Detail & Related papers  (2023-05-22T17:58:04Z)
• What Are You Token About? Dense Retrieval as Distributions Over the Vocabulary [68.77983831618685]
  We propose to interpret the vector representations produced by dual encoders by projecting them into the model's vocabulary space. We show that the resulting projections contain rich semantic information, and draw connection between them and sparse retrieval.
  arXiv  Detail & Related papers  (2022-12-20T16:03:25Z)
• Measuring the Interpretability of Unsupervised Representations via Quantized Reverse Probing [97.70862116338554]
  We investigate the problem of measuring interpretability of self-supervised representations. We formulate the latter as estimating the mutual information between the representation and a space of manually labelled concepts. We use our method to evaluate a large number of self-supervised representations, ranking them by interpretability.
  arXiv  Detail & Related papers  (2022-09-07T16:18:50Z)
• More Than Words: Towards Better Quality Interpretations of Text Classifiers [16.66535643383862]
  We show that token-based interpretability, while being a convenient first choice given the input interfaces of the ML models, is not the most effective one in all situations. We show that higher-level feature attributions offer several advantages: 1) they are more robust as measured by the randomization tests, 2) they lead to lower variability when using approximation-based methods like SHAP, and 3) they are more intelligible to humans in situations where the linguistic coherence resides at a higher level.
  arXiv  Detail & Related papers  (2021-12-23T10:18:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.