Follow-up Attention: An Empirical Study of Developer and Neural Model Code Exploration

• URL: http://arxiv.org/abs/2210.05506v2
• Date: Thu, 29 Aug 2024 14:36:20 GMT
• Title: Follow-up Attention: An Empirical Study of Developer and Neural Model Code Exploration
• Authors: Matteo Paltenghi, Rahul Pandita, Austin Z. Henley, Albert Ziegler,
• Abstract summary: Recent neural models of code, such as OpenAI Codex and AlphaCode, have demonstrated remarkable proficiency at code generation. However, it often remains unclear how the models actually process code, and to what extent their reasoning and the way their attention mechanism scans the code matches the patterns of developers. This work studies how the processed attention signal of three open large language models - CodeGen, InCoder and GPT-J - agrees with how developers look at and explore code.
• Score: 6.060235526273212
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent neural models of code, such as OpenAI Codex and AlphaCode, have demonstrated remarkable proficiency at code generation due to the underlying attention mechanism. However, it often remains unclear how the models actually process code, and to what extent their reasoning and the way their attention mechanism scans the code matches the patterns of developers. A poor understanding of the model reasoning process limits the way in which current neural models are leveraged today, so far mostly for their raw prediction. To fill this gap, this work studies how the processed attention signal of three open large language models - CodeGen, InCoder and GPT-J - agrees with how developers look at and explore code when each answers the same sensemaking questions about code. Furthermore, we contribute an open-source eye-tracking dataset comprising 92 manually-labeled sessions from 25 developers engaged in sensemaking tasks. We empirically evaluate five heuristics that do not use the attention and ten attention-based post-processing approaches of the attention signal of CodeGen against our ground truth of developers exploring code, including the novel concept of follow-up attention which exhibits the highest agreement between model and human attention. Our follow-up attention method can predict the next line a developer will look at with 47% accuracy. This outperforms the baseline prediction accuracy of 42.3%, which uses the session history of other developers to recommend the next line. These results demonstrate the potential of leveraging the attention signal of pre-trained models for effective code exploration.

Related papers

• OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critique [59.18475981916166]
  We introduce OpenCodeReasoning-II, a dataset consisting of 2.5M question-solution-critique triples (approx. 35K unique programming questions)<n>In this work, we employ a two-stage supervised fine-tuning strategy. The first stage focuses on fine-tuning for code generation, while the second stage involves the joint training of models for both code generation and critique. Notably, the integration of our code generation and critique models leads to significant improvements in competitive coding performance.
  arXiv  Detail & Related papers  (2025-07-11T23:35:54Z)
• Automatic Bias Detection in Source Code Review [2.3480418671346164]
  We propose a controlled experiment to detect potentially biased outcomes in code reviews by observing how reviewers interact with the code. We employ the "spotlight model of attention", a cognitive framework where a reviewer's gaze is tracked to determine their focus areas on the review screen. We plan to analyze the sequence of gaze focus using advanced sequence modeling techniques, including Markov Models, Recurrent Neural Networks (RNNs), and Conditional Random Fields (CRF)
  arXiv  Detail & Related papers  (2025-04-25T16:01:52Z)
• Meta-Representational Predictive Coding: Biomimetic Self-Supervised Learning [51.22185316175418]
  We present a new form of predictive coding that we call meta-representational predictive coding (MPC) MPC sidesteps the need for learning a generative model of sensory input by learning to predict representations of sensory input across parallel streams.
  arXiv  Detail & Related papers  (2025-03-22T22:13:14Z)
• Enhancing Code LLM Training with Programmer Attention [11.622059894637683]
  We introduce an eye-tracking path augmentation method to expand programmer attention datasets. We also introduce a pattern abstraction step that refines raw fixations into learnable attention motifs. Our experiments yield +7.16 in CodeBLEU on the CodeXGlue benchmark for code summarization.
  arXiv  Detail & Related papers  (2025-03-19T06:44:29Z)
• EnseSmells: Deep ensemble and programming language models for automated code smells detection [3.974095344344234]
  A smell in software source code denotes an indication of suboptimal design and implementation decisions. This paper proposes a novel approach to code smell detection, constructing a deep learning architecture that places importance on the fusion of structural features and statistical semantics.
  arXiv  Detail & Related papers  (2025-02-07T15:35:19Z)
• Prompting and Fine-tuning Large Language Models for Automated Code Review Comment Generation [5.6001617185032595]
  Large language models pretrained on both programming and natural language data tend to perform well in code-oriented tasks. We fine-tune open-source Large language models (LLM) in parameter-efficient, quantized low-rank fashion on consumer-grade hardware to improve review comment generation.
  arXiv  Detail & Related papers  (2024-11-15T12:01:38Z)
• Understanding Code Understandability Improvements in Code Reviews [79.16476505761582]
  We analyzed 2,401 code review comments from Java open-source projects on GitHub. 83.9% of suggestions for improvement were accepted and integrated, with fewer than 1% later reverted.
  arXiv  Detail & Related papers  (2024-10-29T12:21:23Z)
• VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning [59.68917139718813]
  We show that a strong off-the-shelf frozen pretrained visual encoder can achieve state-of-the-art (SoTA) performance in forecasting and procedural planning. By conditioning on frozen clip-level embeddings from observed steps to predict the actions of unseen steps, our prediction model is able to learn robust representations for forecasting.
  arXiv  Detail & Related papers  (2024-10-04T14:52:09Z)
• Toward Exploring the Code Understanding Capabilities of Pre-trained Code Generation Models [12.959392500354223]
  We pioneer the transfer of knowledge from pre-trained code generation models to code understanding tasks. We introduce CL4D, a contrastive learning method designed to enhance the representation capabilities of decoder-only models.
  arXiv  Detail & Related papers  (2024-06-18T06:52:14Z)
• Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach [66.51005288743153]
  We investigate the legal and ethical issues of current neural code completion models. We tailor a membership inference approach (termed CodeMI) that was originally crafted for classification tasks. We evaluate the effectiveness of this adapted approach across a diverse array of neural code completion models.
  arXiv  Detail & Related papers  (2024-04-22T15:54:53Z)
• Naturalness of Attention: Revisiting Attention in Code Language Models [3.756550107432323]
  Language models for code such as CodeBERT offer the capability to learn advanced source code representation, but their opacity poses barriers to understanding of captured properties. This study aims to shed some light on the previously ignored factors of the attention mechanism beyond the attention weights.
  arXiv  Detail & Related papers  (2023-11-22T16:34:12Z)
• Towards Modeling Human Attention from Eye Movements for Neural Source Code Summarization [6.435578628605734]
  We use eye-tracking data to create a model of human attention. The model predicts which words in source code are the most important for code summarization. We observe an improvement in prediction performance of the augmented approach in line with other bio-inspired neural models.
  arXiv  Detail & Related papers  (2023-05-16T19:56:45Z)
• Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions [54.55334589363247]
  We study whether conveying information about uncertainty enables programmers to more quickly and accurately produce code. We find that highlighting tokens with the highest predicted likelihood of being edited leads to faster task completion and more targeted edits.
  arXiv  Detail & Related papers  (2023-02-14T18:43:34Z)
• Data-Driven and SE-assisted AI Model Signal-Awareness Enhancement and Introspection [61.571331422347875]
  We propose a data-driven approach to enhance models' signal-awareness. We combine the SE concept of code complexity with the AI technique of curriculum learning. We achieve up to 4.8x improvement in model signal awareness.
  arXiv  Detail & Related papers  (2021-11-10T17:58:18Z)
• Bayesian Attention Belief Networks [59.183311769616466]
  Attention-based neural networks have achieved state-of-the-art results on a wide range of tasks. This paper introduces Bayesian attention belief networks, which construct a decoder network by modeling unnormalized attention weights. We show that our method outperforms deterministic attention and state-of-the-art attention in accuracy, uncertainty estimation, generalization across domains, and adversarial attacks.
  arXiv  Detail & Related papers  (2021-06-09T17:46:22Z)
• Demystifying Code Summarization Models [5.608277537412537]
  We evaluate four prominent code summarization models: extreme summarizer, code2vec, code2seq, and sequence GNN. Results show that all models base their predictions on syntactic and lexical properties with little to none semantic implication. We present a novel approach to explaining the predictions of code summarization models through the lens of training data.
  arXiv  Detail & Related papers  (2021-02-09T03:17:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.