Exploring Language Model's Code Generation Ability with Auxiliary Functions

• URL: http://arxiv.org/abs/2403.10575v1
• Date: Fri, 15 Mar 2024 04:41:50 GMT
• Title: Exploring Language Model's Code Generation Ability with Auxiliary Functions
• Authors: Seonghyeon Lee, Sanghwan Jang, Seongbo Jang, Dongha Lee, Hwanjo Yu,
• Abstract summary: We comprehensively evaluate the ability to utilize auxiliary functions encoded in recent code-pretrained language models. Our analysis reveals the model's underutilized behavior to call the auxiliary function.
• Score: 25.951695685705637
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Auxiliary function is a helpful component to improve language model's code generation ability. However, a systematic exploration of how they affect has yet to be done. In this work, we comprehensively evaluate the ability to utilize auxiliary functions encoded in recent code-pretrained language models. First, we construct a human-crafted evaluation set, called HumanExtension, which contains examples of two functions where one function assists the other. With HumanExtension, we design several experiments to examine their ability in a multifaceted way. Our evaluation processes enable a comprehensive understanding of including auxiliary functions in the prompt in terms of effectiveness and robustness. An additional implementation style analysis captures the models' various implementation patterns when they access the auxiliary function. Through this analysis, we discover the models' promising ability to utilize auxiliary functions including their self-improving behavior by implementing the two functions step-by-step. However, our analysis also reveals the model's underutilized behavior to call the auxiliary function, suggesting the future direction to enhance their implementation by eliciting the auxiliary function call ability encoded in the models. We release our code and dataset to facilitate this research direction.

Related papers

• Function Induction and Task Generalization: An Interpretability Study with Off-by-One Addition [52.11481619456093]
  We find a function induction mechanism that explains the model's generalization from standard addition to off-by-one addition.<n>This mechanism resembles the structure of the induction head mechanism found in prior work and elevates it to a higher level of abstraction.<n>We find that this function induction mechanism is reused in a broader range of tasks, including synthetic tasks such as shifted multiple-choice QA and algorithmic tasks such as base-8 addition.
  arXiv  Detail & Related papers  (2025-07-14T03:20:55Z)
• Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation [85.68881632498909]
  We propose a principled framework for synthesizing high-quality training trajectories for large language model agents. The framework is based on automatic and iterative translations from a function signature path to a sequence of queries and executable function calls. Experiments show that training with the positive trajectories with supervised fine-tuning and preference optimization against negative trajectories, our 14B model, Magnet-14B-mDPO, obtains 68.01 on BFCL-v3 and 73.30 on ToolQuery.
  arXiv  Detail & Related papers  (2025-03-10T20:13:07Z)
• Eliciting Instruction-tuned Code Language Models' Capabilities to Utilize Auxiliary Function for Code Generation [25.434546255499242]
  We study the code generation behavior of instruction-tuned models built on top of code pre-trained language models. We design several ways to provide auxiliary functions to the models by adding them to the query or providing a response prefix.
  arXiv  Detail & Related papers  (2024-09-20T22:28:20Z)
• Adaptive Language-Guided Abstraction from Contrastive Explanations [53.48583372522492]
  It is necessary to determine which features of the environment are relevant before determining how these features should be used to compute reward. End-to-end methods for joint feature and reward learning often yield brittle reward functions that are sensitive to spurious state features. This paper describes a method named ALGAE which alternates between using language models to iteratively identify human-meaningful features.
  arXiv  Detail & Related papers  (2024-09-12T16:51:58Z)
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models [55.19497659895122]
  We introduce methods for discovering and applying sparse feature circuits. These are causally implicatedworks of human-interpretable features for explaining language model behaviors.
  arXiv  Detail & Related papers  (2024-03-28T17:56:07Z)
• FIND: A Function Description Benchmark for Evaluating Interpretability Methods [86.80718559904854]
  This paper introduces FIND (Function INterpretation and Description), a benchmark suite for evaluating automated interpretability methods. FIND contains functions that resemble components of trained neural networks, and accompanying descriptions of the kind we seek to generate. We evaluate methods that use pretrained language models to produce descriptions of function behavior in natural language and code.
  arXiv  Detail & Related papers  (2023-09-07T17:47:26Z)
• A Functional approach for Two Way Dimension Reduction in Time Series [13.767812547998735]
  We propose a non-linear function-on-function approach, which consists of a functional encoder and a functional decoder. Our approach gives a low dimension latent representation by reducing the number of functional features as well as the timepoints at which the functions are observed.
  arXiv  Detail & Related papers  (2023-01-01T06:09:15Z)
• Latent SHAP: Toward Practical Human-Interpretable Explanations [44.28376542666342]
  We introduce Latent SHAP, a black-box feature attribution framework that provides human-interpretable explanations. We demonstrate Latent SHAP's effectiveness using (1) a controlled experiment where invertible transformation functions are available, which enables robust quantitative evaluation of our method, and (2) celebrity attractiveness classification (using the CelebA dataset) where invertible transformation functions are not available.
  arXiv  Detail & Related papers  (2022-11-27T11:33:26Z)
• Neural Estimation of Submodular Functions with Applications to Differentiable Subset Selection [50.14730810124592]
  Submodular functions and variants, through their ability to characterize diversity and coverage, have emerged as a key tool for data selection and summarization. We propose FLEXSUBNET, a family of flexible neural models for both monotone and non-monotone submodular functions.
  arXiv  Detail & Related papers  (2022-10-20T06:00:45Z)
• Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficient [65.08966446962845]
  offline reinforcement learning, which aims at optimizing decision-making strategies with historical data, has been extensively applied in real-life applications. We take a step by considering offline reinforcement learning with differentiable function class approximation (DFA) Most importantly, we show offline differentiable function approximation is provably efficient by analyzing the pessimistic fitted Q-learning algorithm.
  arXiv  Detail & Related papers  (2022-10-03T07:59:42Z)
• Neural Network Approximation of Refinable Functions [8.323468006516018]
  We show that refinable functions are approximated by the outputs of deep ReLU networks with a fixed width and increasing depth with accuracy exponential. Our results apply to functions used in the standard construction of wavelets as well as to functions constructed via subdivision algorithms in Computer Aided Geometric Design.
  arXiv  Detail & Related papers  (2021-07-28T06:45:36Z)
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions [55.660255727031725]
  Influence functions explain the decisions of a model by identifying influential training examples. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. We develop a new measure based on influence functions that can reveal artifacts in training data.
  arXiv  Detail & Related papers  (2020-05-14T00:45:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.