Bringing Structure to Naturalness: On the Naturalness of ASTs

• URL: http://arxiv.org/abs/2504.08234v1
• Date: Fri, 11 Apr 2025 03:43:46 GMT
• Title: Bringing Structure to Naturalness: On the Naturalness of ASTs
• Authors: Profir-Petru Pârţachi, Mahito Sugiyama,
• Abstract summary: We show that structured representations of code are similarly statistically predictable, i.e. that a structured view of code is also natural.<n>We show how such naturalness signals can be employed for near state-of-the-art results on just-in-time defect prediction.
• Score: 9.100580570005407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Source code comes in different shapes and forms. Previous research has already shown code to be more predictable than natural language as well as highlighted its statistical predictability at the token level: source code can be natural. More recently, the structure of code -- control flow, syntax graphs, abstract syntax trees etc. -- has been successfully used to improve the state-of-the-art on numerous tasks: code suggestion, code summarisation, method naming etc. This body of work implicitly assumes that structured representations of code are similarly statistically predictable, i.e. that a structured view of code is also natural. We consider that this view should be made explicit and propose directly studying the Structured Naturalness Hypothesis. Beyond just naming existing research that assumes this hypothesis and formulating it, we also provide evidence in the case of trees: TreeLSTM models over ASTs for some languages, such as Ruby, are competitive with $n$-gram models while handling the syntax token issue highlighted by previous research 'for free'. For other languages, such as Java or Python, we find tree models to perform worse, suggesting that downstream task improvement is uncorrelated to the language modelling task. Further, we show how such naturalness signals can be employed for near state-of-the-art results on just-in-time defect prediction while forgoing manual feature engineering work.

Related papers

• CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction [47.17755403213469]
  We propose CodeI/O, a novel approach that condenses diverse reasoning patterns embedded in contextually-grounded codes.<n>By training models to predict inputs/outputs given code and test cases entirely in natural language, we expose them to universal reasoning primitives.<n> Experimental results demonstrate CodeI/O leads to consistent improvements across symbolic, scientific, logic, math & numerical, and commonsense reasoning tasks.
  arXiv  Detail & Related papers  (2025-02-11T07:26:50Z)
• Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning [53.241569810013836]
  We propose a novel framework that utilizes large language models (LLMs) to identify effective feature generation rules. We use decision trees to convey this reasoning information, as they can be easily represented in natural language. OCTree consistently enhances the performance of various prediction models across diverse benchmarks.
  arXiv  Detail & Related papers  (2024-06-12T08:31:34Z)
• SCALE: Constructing Structured Natural Language Comment Trees for Software Vulnerability Detection [36.37244302912536]
  We propose a Structured Natural Language Comment tree-based vulnerAbiLity dEtection framework based on the pre-trained models. The proposed Structured Natural Language Comment Tree (SCT) integrates the semantics of code statements with code execution sequences.
  arXiv  Detail & Related papers  (2024-03-28T02:20:03Z)
• CodeKGC: Code Language Model for Generative Knowledge Graph Construction [46.220237225553234]
  Large generative language model trained on structured data such as code has demonstrated impressive capability in understanding natural language for structural prediction and reasoning tasks. We develop schema-aware prompts that effectively utilize the semantic structure within the knowledge graph. Experimental results indicate that the proposed approach can obtain better performance on benchmark datasets compared with baselines.
  arXiv  Detail & Related papers  (2023-04-18T15:12:34Z)
• Python Code Generation by Asking Clarification Questions [57.63906360576212]
  In this work, we introduce a novel and more realistic setup for this task. We hypothesize that the under-specification of a natural language description can be resolved by asking clarification questions. We collect and introduce a new dataset named CodeClarQA containing pairs of natural language descriptions and code with created synthetic clarification questions and answers.
  arXiv  Detail & Related papers  (2022-12-19T22:08:36Z)
• Benchmarking Language Models for Code Syntax Understanding [79.11525961219591]
  Pre-trained language models have demonstrated impressive performance in both natural language processing and program understanding. In this work, we perform the first thorough benchmarking of the state-of-the-art pre-trained models for identifying the syntactic structures of programs. Our findings point out key limitations of existing pre-training methods for programming languages, and suggest the importance of modeling code syntactic structures.
  arXiv  Detail & Related papers  (2022-10-26T04:47:18Z)
• Unnatural Language Inference [48.45003475966808]
  We find that state-of-the-art NLI models, such as RoBERTa and BART, are invariant to, and sometimes even perform better on, examples with randomly reordered words. Our findings call into question the idea that our natural language understanding models, and the tasks used for measuring their progress, genuinely require a human-like understanding of syntax.
  arXiv  Detail & Related papers  (2020-12-30T20:40:48Z)
• Exploring Software Naturalness through Neural Language Models [56.1315223210742]
  The Software Naturalness hypothesis argues that programming languages can be understood through the same techniques used in natural language processing. We explore this hypothesis through the use of a pre-trained transformer-based language model to perform code analysis tasks.
  arXiv  Detail & Related papers  (2020-06-22T21:56:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.