Read between the lines -- Functionality Extraction From READMEs

• URL: http://arxiv.org/abs/2403.10205v1
• Date: Fri, 15 Mar 2024 11:11:57 GMT
• Title: Read between the lines -- Functionality Extraction From READMEs
• Abstract summary: We introduce a novel and useful variant of it called functionality extraction from Git files. The motivation behind this task stems from a recent surge in research and development activities around the use of large language models for code-related tasks. Our best fine-tuned 7 Billion CodeLlama model exhibit 70% and 20% gain on the F1 score against ChatGPT and Bard respectively.
• Score: 7.302971710984904
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While text summarization is a well-known NLP task, in this paper, we introduce a novel and useful variant of it called functionality extraction from Git README files. Though this task is a text2text generation at an abstract level, it involves its own peculiarities and challenges making existing text2text generation systems not very useful. The motivation behind this task stems from a recent surge in research and development activities around the use of large language models for code-related tasks, such as code refactoring, code summarization, etc. We also release a human-annotated dataset called FuncRead, and develop a battery of models for the task. Our exhaustive experimentation shows that small size fine-tuned models beat any baseline models that can be designed using popular black-box or white-box large language models (LLMs) such as ChatGPT and Bard. Our best fine-tuned 7 Billion CodeLlama model exhibit 70% and 20% gain on the F1 score against ChatGPT and Bard respectively.

Related papers

• BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion [52.26069987232466]
  We propose BLooP, a simple training-free decoding intervention that encourages large language models to generate bigrams from the source document.<n>BLooP operates through a hash table lookup at each decoding step, requiring no training, fine-tuning, or model modification.<n>Human evaluation shows that BLooP significantly improves faithfulness without reducing readability.
  arXiv  Detail & Related papers  (2026-03-12T01:14:19Z)
• DPS: Design Pattern Summarisation Using Code Features [8.24515384844758]
  We generate summaries for software design patterns using Java and NLG libraries. Our summaries closely align with human-written summaries. A follow-up survey shows that DPS summaries were rated as capturing context better than human-generated summaries.
  arXiv  Detail & Related papers  (2025-04-15T11:27:44Z)
• FeRG-LLM : Feature Engineering by Reason Generation Large Language Models [2.6740666148510077]
  FeRG-LLM is a large language model designed to automatically perform feature engineering. We have constructed two-stage conversational dialogues that enable language models to analyze machine learning tasks. Experiments show that FeRG-LLM performs comparably to or better than Llama 3.1 70B on most datasets.
  arXiv  Detail & Related papers  (2025-03-30T09:07:21Z)
• Idiosyncrasies in Large Language Models [54.26923012617675]
  We unveil and study idiosyncrasies in Large Language Models (LLMs) We find that fine-tuning existing text embedding models on LLM-generated texts yields excellent classification accuracy. We leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies.
  arXiv  Detail & Related papers  (2025-02-17T18:59:02Z)
• Generative Representational Instruction Tuning [89.76840377003178]
  GritLM 7B sets a new state of the art on the Massive Text Embedding Benchmark (MTEB) GritLM 8x7B outperforms all open generative language models that we tried while still being among the best embedding models.
  arXiv  Detail & Related papers  (2024-02-15T12:12:19Z)
• Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models [0.0]
  Leveraging Large Language Models (LLMs) has shown remarkable promise in enhancing summarization techniques. This paper embarks on an exploration of text summarization with a diverse set of LLMs, including MPT-7b-instruct, falcon-7b-instruct, and OpenAI ChatGPT text-davinci-003 models.
  arXiv  Detail & Related papers  (2023-10-16T14:33:02Z)
• Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphs [4.56877715768796]
  We show that ChatGPT achieves near state-of-the-art performance on some measures of the WebNLG 2020 challenge. We also show that there is a significant connection between what the LLM already knows about the data it is parsing and the quality of the output text.
  arXiv  Detail & Related papers  (2023-07-14T12:45:03Z)
• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [51.90524745663737]
  A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks. Our method achieves state-of-the-art results on well-established TAG datasets. Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
  arXiv  Detail & Related papers  (2023-05-31T03:18:03Z)
• Understanding the Effectiveness of Very Large Language Models on Dialog Evaluation [20.18656308749408]
  Large language models (LLMs) have been used for generation and can now output human-like text. This paper investigates how the number of examples in the prompt and the type of example selection used affect the model's performance.
  arXiv  Detail & Related papers  (2023-01-27T22:02:27Z)
• Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning [53.92465205531759]
  Controlled automated story generation seeks to generate natural language stories satisfying constraints from natural language critiques or preferences. We train a contrastive bi-encoder model to align stories with human critiques, building a general purpose preference model. We further fine-tune the contrastive reward model using a prompt-learning technique to increase story generation robustness.
  arXiv  Detail & Related papers  (2022-10-14T13:21:33Z)
• On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization [89.94078728495423]
  We show that recent advances in each modality, CLIP image representations and scaling of language models, do not consistently improve multimodal self-rationalization of tasks with multimodal inputs. Our findings call for a backbone modelling approach that can be built on to advance text generation from images and text beyond image captioning.
  arXiv  Detail & Related papers  (2022-05-24T00:52:40Z)
• KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation [100.79870384880333]
  We propose a knowledge-grounded pre-training (KGPT) to generate knowledge-enriched text. We adopt three settings, namely fully-supervised, zero-shot, few-shot to evaluate its effectiveness. Under zero-shot setting, our model achieves over 30 ROUGE-L on WebNLG while all other baselines fail.
  arXiv  Detail & Related papers  (2020-10-05T19:59:05Z)
• Language Models are Few-Shot Learners [61.36677350504291]
  We show that scaling up language models greatly improves task-agnostic, few-shot performance. We train GPT-3, an autoregressive language model with 175 billion parameters, and test its performance in the few-shot setting. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks.
  arXiv  Detail & Related papers  (2020-05-28T17:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.