Large Language Models Should Ask Clarifying Questions to Increase Confidence in Generated Code

• URL: http://arxiv.org/abs/2308.13507v2
• Date: Mon, 22 Jan 2024 18:54:52 GMT
• Title: Large Language Models Should Ask Clarifying Questions to Increase Confidence in Generated Code
• Authors: Jie JW Wu
• Abstract summary: Large language models (LLMs) have significantly improved the ability to perform tasks in the field of code generation. There is still a gap between LLMs being capable coders and being top-tier software engineers. I propose a communication-centered process that uses an LLM-generated communicator to identify issues with high ambiguity or low confidence in problem descriptions and generated code.
• Score: 0.7252027234425334
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have significantly improved the ability to perform tasks in the field of code generation. However, there is still a gap between LLMs being capable coders and being top-tier software engineers. Based on the observation that toplevel software engineers often ask clarifying questions to reduce ambiguity in both requirements and coding solutions, I argue that the same should be applied to LLMs for code generation tasks. By asking probing questions in various topics before generating the final code, the challenges of programming with LLMs, such as unclear intent specification, lack of computational thinking, and undesired code quality, may be alleviated. This, in turn, increases confidence in the generated code. In this work, I explore how to leverage better communication skills to achieve greater confidence in generated code. I propose a communication-centered process that uses an LLM-generated communicator to identify issues with high ambiguity or low confidence in problem descriptions and generated code. I then ask clarifying questions to obtain responses from users for refining the code.

Related papers

• Steering Large Language Models between Code Execution and Textual Reasoning [22.279107036500083]
  Textual reasoning has inherent limitations in solving tasks with challenges in math, logics, optimization, and searching. The recently released OpenAI GPT Code Interpreter and multi-agent frameworks such as AutoGen have demonstrated remarkable proficiency of integrating code generation and execution. We propose three methods to better steer LLM code/text generation and achieve a notable improvement.
  arXiv  Detail & Related papers  (2024-10-04T15:44:47Z)
• Beyond Code Generation: Assessing Code LLM Maturity with Postconditions [9.521621889147362]
  We propose a code Large Language Model maturity model based on the postcondition generation problem. We augment the EvalPlus dataset to a postcondition testing benchmark, and evaluate several open-sourced models.
  arXiv  Detail & Related papers  (2024-07-19T08:34:30Z)
• Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent [2.8391355909797644]
  Large language models (LLMs) have significantly improved their ability to perform tasks in the field of code generation. There is still a gap between LLMs being capable coders and being top-tier software engineers.
  arXiv  Detail & Related papers  (2024-05-31T22:06:18Z)
• InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models [56.723509505549536]
  InfiBench is the first large-scale freeform question-answering (QA) benchmark for code to our knowledge. It comprises 234 carefully selected high-quality Stack Overflow questions that span across 15 programming languages. We conduct a systematic evaluation for over 100 latest code LLMs on InfiBench, leading to a series of novel and insightful findings.
  arXiv  Detail & Related papers  (2024-03-11T02:06:30Z)
• Assured LLM-Based Software Engineering [51.003878077888686]
  This paper is an outline of the content of the keynote by Mark Harman at the International Workshop on Interpretability, Robustness, and Benchmarking in Neural Software Engineering, Monday 15th April 2024, Lisbon, Portugal.
  arXiv  Detail & Related papers  (2024-02-06T20:38:46Z)
• Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs [65.2379940117181]
  We introduce code prompting, a chain of prompts that transforms a natural language problem into code. We find that code prompting exhibits a high-performance boost for multiple LLMs. Our analysis of GPT 3.5 reveals that the code formatting of the input problem is essential for performance improvement.
  arXiv  Detail & Related papers  (2024-01-18T15:32:24Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• Test-Case-Driven Programming Understanding in Large Language Models for Better Code Generation [15.166827643436346]
  muFiX is a novel prompting technique to improve the code generation performance of large language models (LLMs) It first exploits test case analysis to obtain specification understanding and enables a self-improvement process. muFiX further fixes the specification understanding towards the direction reducing the gap between the provided understanding and the actual understanding.
  arXiv  Detail & Related papers  (2023-09-28T02:58:07Z)
• Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation [8.575560293086289]
  Large language models (LLMs) have shown extraordinary ability in understanding natural language and generating programming code. The misuse of APIs in the generated code could lead to severe problem, such as resource leaks, program crashes.
  arXiv  Detail & Related papers  (2023-08-20T18:36:28Z)
• CodeT5+: Open Code Large Language Models for Code Understanding and Generation [72.1638273937025]
  Large language models (LLMs) pretrained on vast source code have achieved prominent progress in code intelligence. CodeT5+ is a family of encoder-decoder LLMs for code in which component modules can be flexibly combined to suit a wide range of downstream code tasks. We extensively evaluate CodeT5+ on over 20 code-related benchmarks in different settings, including zero-shot, finetuning, and instruction-tuning.
  arXiv  Detail & Related papers  (2023-05-13T14:23:07Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.