Verifier Warnings Do Not Improve Comprehensibility Prediction

• URL: http://arxiv.org/abs/2604.22653v1
• Date: Fri, 24 Apr 2026 15:28:42 GMT
• Title: Verifier Warnings Do Not Improve Comprehensibility Prediction
• Abstract summary: We show that combining syntactic and developer features is just as effective for predicting human-judged code comprehensibility.<n>Our findings suggest that while a correlation exists, the verifier warning sum offers limited discriminative power.
• Score: 5.123298347655088
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Proponents of software verification suggest that code simplicity is linked to the effort to verify code, hypothesizing that formal verifiers produce fewer false positive warnings and require less manual intervention when analyzing simpler code. A recent meta-analysis study found empirical support for this hypothesis: a small correlation between the sum of verifier warnings and human-derived code comprehensibility metrics. Based on this finding, we conjectured that using the sum of verifier tool (verifier) warnings to represent program semantic information as an input feature to machine learning (ML) models for code comprehensibility prediction can enhance their performance, when combined with traditional syntactic and developer features. To test this conjecture, we performed a control-treatment experiment incorporating the verifier warning sum feature into machine learning models from the literature, and conducted a comparative analysis of their performance against models trained only on syntactic and developer features. We found no significant difference in the prediction performance of models with and without the warnings feature. Our findings suggest that while a correlation exists, the verifier warning sum offers limited discriminative power: combining syntactic and developer features is just as effective for predicting human-judged code comprehensibility.

Related papers

• Evaluating Line-level Localization Ability of Learning-based Code Vulnerability Detection Models [9.543689542888599]
  We propose an explainability-based evaluation procedure for vulnerability detectors.<n>Our approach, defined as Detection Alignment (DA), quantifies the agreement between the input source code lines.<n>We show how the predictions of such models are consistently biased by non-vulnerable lines.
  arXiv  Detail & Related papers  (2025-10-13T09:34:40Z)
• Relative Code Comprehensibility Prediction [5.123298347655088]
  We conduct a study to assess and compare the effectiveness of absolute and relative code comprehensibility prediction via machine learning.<n>Our findings indicate that absolute comprehensibility models improve over the baselines by at most 33.4% and frequently underperform.<n>Relative comprehensibility models are substantially better, with average improvements of 137.8% and 74.7% for snippet-wise prediction.
  arXiv  Detail & Related papers  (2025-10-03T19:51:43Z)
• Towards Automated Error Discovery: A Study in Conversational AI [48.735443116662026]
  We introduce Automated Error Discovery, a framework for detecting and defining errors in conversational AI.<n>We also propose SEEED (Soft Clustering Extended-Based Error Detection), as an encoder-based approach to its implementation.
  arXiv  Detail & Related papers  (2025-09-13T14:53:22Z)
• To Err is Machine: Vulnerability Detection Challenges LLM Reasoning [8.602355712876815]
  We present a challenging code reasoning task: vulnerability detection.<n>State-of-the-art (SOTA) models reported only 54.5% Balanced Accuracy in our vulnerability detection evaluation.<n>New models, new training methods, or more execution-specific pretraining data may be needed to conquer vulnerability detection.
  arXiv  Detail & Related papers  (2024-03-25T21:47:36Z)
• On the Relationship between Code Verifiability and Understandability [2.5728707125824735]
  Proponents of software verification have argued that simpler code is easier to verify. We compare the number of warnings produced by four state-of-the-art verification tools on 211 snippets of Java code with 20 metrics of code comprehensibility from human subjects.
  arXiv  Detail & Related papers  (2023-10-31T03:54:35Z)
• Controlling Learned Effects to Reduce Spurious Correlations in Text Classifiers [6.662800021628275]
  We propose an algorithm to regularize the learnt effect of the features on the model's prediction to the estimated effect of feature on label. On toxicity and IMDB review datasets, the proposed algorithm minimises spurious correlations and improves the minority group.
  arXiv  Detail & Related papers  (2023-05-26T12:15:54Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)
• Pathologies of Pre-trained Language Models in Few-shot Fine-tuning [50.3686606679048]
  We show that pre-trained language models with few examples show strong prediction bias across labels. Although few-shot fine-tuning can mitigate the prediction bias, our analysis shows models gain performance improvement by capturing non-task-related features. These observations alert that pursuing model performance with fewer examples may incur pathological prediction behavior.
  arXiv  Detail & Related papers  (2022-04-17T15:55:18Z)
• Prototypical Representation Learning for Relation Extraction [56.501332067073065]
  This paper aims to learn predictive, interpretable, and robust relation representations from distantly-labeled data. We learn prototypes for each relation from contextual information to best explore the intrinsic semantics of relations. Results on several relation learning tasks show that our model significantly outperforms the previous state-of-the-art relational models.
  arXiv  Detail & Related papers  (2021-03-22T08:11:43Z)
• Adversarial Examples for Unsupervised Machine Learning Models [71.81480647638529]
  Adrial examples causing evasive predictions are widely used to evaluate and improve the robustness of machine learning models. We propose a framework of generating adversarial examples for unsupervised models and demonstrate novel applications to data augmentation.
  arXiv  Detail & Related papers  (2021-03-02T17:47:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.