Protocode: Prototype-Driven Interpretability for Code Generation in LLMs

• URL: http://arxiv.org/abs/2509.25247v1
• Date: Sat, 27 Sep 2025 00:32:45 GMT
• Title: Protocode: Prototype-Driven Interpretability for Code Generation in LLMs
• Authors: Krishna Vamshi Bodla, Haizhao Yang,
• Abstract summary: Large Language Models (LLMs) have been widely adopted for various tasks such as text summarization, question answering, speech-to-text translation, and more.<n>Our work focuses on automatically sampling In-Context Learning (ICL) demonstrations which can improve model performance and enhance the interpretability of the generated code.
• Score: 5.8296917468117835
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Since the introduction of Large Language Models (LLMs), they have been widely adopted for various tasks such as text summarization, question answering, speech-to-text translation, and more. In recent times, the use of LLMs for code generation has gained significant attention, with tools such as Cursor and Windsurf demonstrating the ability to analyze massive code repositories and recommend relevant changes. Big tech companies have also acknowledged the growing reliance on LLMs for code generation within their codebases. Although these advances significantly improve developer productivity, increasing reliance on automated code generation can proportionally increase the risk of suboptimal solutions and insecure code. Our work focuses on automatically sampling In-Context Learning (ICL) demonstrations which can improve model performance and enhance the interpretability of the generated code. Using AST-based analysis on outputs from the MBPP test set, we identify regions of code most influenced by the chosen demonstrations. In our experiments, we show that high-quality ICL demonstrations not only make outputs easier to interpret but also yield a positive performance improvement on the pass@10 metric. Conversely, poorly chosen ICL demonstrations affected the LLM performance on the pass@10 metric negatively compared to the base model. Overall, our approach highlights the importance of efficient sampling strategies for ICL, which can affect the performance of the model on any given task.

Related papers

• What Builds Effective In-Context Examples for Code Generation? [8.536350550103057]
  In-Context Learning (ICL) has emerged as a promising solution to enhance the code generation capabilities of Large Language Models (LLMs)<n>This paper systematically investigates the impact of various code features on ICL with code examples through controlled ablation studies.
  arXiv  Detail & Related papers  (2025-08-08T15:58:11Z)
• Assessing Small Language Models for Code Generation: An Empirical Study with Benchmarks [4.448709087838503]
  Small Language Models (SLMs) offer lightweight and cost-effective alternatives to Large Language Models (LLMs)<n>This study presents a comprehensive empirical evaluation of 20 open-source SLMs ranging from 0.4B to 10B parameters on five code-related benchmarks.
  arXiv  Detail & Related papers  (2025-07-03T20:32:36Z)
• Evaluating Large Language Models on Non-Code Software Engineering Tasks [4.381476817430934]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code understanding and generation.<n>We present the first comprehensive benchmark, which we name Software Engineering Language Understanding' (SELU)<n>SELU covers classification, regression, Named Entity Recognition (NER) and Masked Language Modeling (MLM) targets, with data drawn from diverse sources.
  arXiv  Detail & Related papers  (2025-06-12T15:52:32Z)
• Prompt engineering and framework: implementation to increase code reliability based guideline for LLMs [0.0]
  We introduce a prompt template designed to improve the quality and correctness of generated code snippets.<n>We demonstrate that our approach outperforms widely studied zero-shot and Chain-of-Thought (CoT) methods in terms of the Pass@k metric.
  arXiv  Detail & Related papers  (2025-03-19T18:33:08Z)
• Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding [71.01099784480597]
  Large language models (LLMs) excel at a range of tasks through in-context learning (ICL)<n>We introduce In-Context Contrastive Decoding (ICCD), a novel method that emphasizes input-label mapping.
  arXiv  Detail & Related papers  (2025-02-19T14:04:46Z)
• A Performance Study of LLM-Generated Code on Leetcode [1.747820331822631]
  This study evaluates the efficiency of code generation by Large Language Models (LLMs) We compare 18 LLMs, considering factors such as model temperature and success rate, and their impact on code performance. We find that LLMs are capable of generating code that is, on average, more efficient than the code written by humans.
  arXiv  Detail & Related papers  (2024-07-31T13:10:03Z)
• Aligning Language Models with Demonstrated Feedback [58.834937450242975]
  Demonstration ITerated Task Optimization (DITTO) directly aligns language model outputs to a user's demonstrated behaviors.<n>We evaluate DITTO's ability to learn fine-grained style and task alignment across domains such as news articles, emails, and blog posts.
  arXiv  Detail & Related papers  (2024-06-02T23:13:56Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs)<n>We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• Exploring Data-Efficient Adaptation of Large Language Models for Code Generation [64.5583894165813]
  We propose a novel adaptation approach named DEED, which stands for Data-Efficient adaptation with Error-Driven learning for code generation.<n> Experimental results show that, compared to other mainstream fine-tuning approaches, DEED achieves superior performance with few training data.
  arXiv  Detail & Related papers  (2024-02-29T16:09:02Z)
• Code Needs Comments: Enhancing Code LLMs with Comment Augmentation [91.52444946362547]
  We introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language. We conducted experiments on three code-focused Large Language Models and observed consistent improvements in performance on two widely-used programming skill benchmarks.
  arXiv  Detail & Related papers  (2024-02-20T13:56:38Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE [62.13435256279566]
  Large Language Models (LLMs) have achieved remarkable performance across a wide variety of natural language tasks. However, their large size makes their inference slow and computationally expensive. We show that it enables these layers to acquire 'good' generation ability without affecting the generation ability of the final layer.
  arXiv  Detail & Related papers  (2023-10-28T04:07:58Z)
• Iterative Forward Tuning Boosts In-Context Learning in Language Models [88.25013390669845]
  In this study, we introduce a novel two-stage framework to boost in-context learning in large language models (LLMs) Specifically, our framework delineates the ICL process into two distinct stages: Deep-Thinking and test stages. The Deep-Thinking stage incorporates a unique attention mechanism, i.e., iterative enhanced attention, which enables multiple rounds of information accumulation.
  arXiv  Detail & Related papers  (2023-05-22T13:18:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.