SmartNote: An LLM-Powered, Personalised Release Note Generator That Just Works

• URL: http://arxiv.org/abs/2505.17977v1
• Date: Fri, 23 May 2025 14:45:44 GMT
• Title: SmartNote: An LLM-Powered, Personalised Release Note Generator That Just Works
• Authors: Farbod Daneshyan, Runzhi He, Jianyu Wu, Minghui Zhou,
• Abstract summary: Many developers view the process of writing software release notes as a tedious and dreadful task.<n>We propose SmartNote, a novel and widely applicable release note generation approach.<n>It produces high-quality, contextually personalised release notes using LLM technology.
• Score: 5.9029064046556545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The release note is a crucial document outlining changes in new software versions. Yet, many developers view the process of writing software release notes as a tedious and dreadful task. Consequently, numerous tools have been developed by researchers and practitioners to automate the generation of software release notes. However, these tools fail to consider project domain and target audience for personalisation, limiting their relevance and conciseness. Additionally, they suffer from limited applicability, often necessitating significant workflow adjustments and adoption efforts, hindering practical use and stressing developers. Despite recent advancements in natural language processing and the proven capabilities of large language models in various code and text-related tasks, there are no existing studies investigating the integration and utilisation of LLMs in automated release note generation. Therefore, we propose SmartNote, a novel and widely applicable release note generation approach that produces high-quality, contextually personalised release notes using LLM technology. SmartNote aggregates changes and uses an LLM to describe and summarise the changes using code, commit, and pull request details. It categorises and scores commits to generate structured and concise release notes of prioritised changes. Our human and automatic evaluations reveal that SmartNote outperforms or achieves comparable performance to DeepRelease, Conventional Changelog, and the projects'original release notes across four quality metrics: completeness, clarity, conciseness, and organisation. In both evaluations, SmartNote ranked first for completeness and organisation, while clarity ranked first in the human evaluation. A further evaluation demonstrates that SmartNote is effective in terms of context awareness and applicability.

Related papers

• Model Editing for New Document Integration in Generative Information Retrieval [110.90609826290968]
  Generative retrieval (GR) reformulates the Information Retrieval (IR) task as the generation of document identifiers (docIDs)<n>Existing GR models exhibit poor generalization to newly added documents, often failing to generate the correct docIDs.<n>We propose DOME, a novel method that effectively and efficiently adapts GR models to unseen documents.
  arXiv  Detail & Related papers  (2026-03-03T09:13:38Z)
• Consistency-Aware Editing for Entity-level Unlearning in Language Models [53.522931419965424]
  We introduce a novel consistency-aware editing (CAE) framework for entity-level unlearning.<n>CAE aggregates a diverse set of prompts related to a target entity, including its attributes, relations, and adversarial paraphrases.<n>It then jointly learns a low-rank update guided by a consistency regularizer that aligns the editing directions across prompts.
  arXiv  Detail & Related papers  (2025-12-19T15:18:07Z)
• ReleaseEval: A Benchmark for Evaluating Language Models in Automated Release Note Generation [20.424587551582153]
  ReleaseEval is a benchmark designed to evaluate language models for automated release note generation.<n>It comprises 94,987 release notes from 3,369 repositories across 6 programming languages.<n>Both automated and human evaluations show that large language models consistently outperform traditional baselines.
  arXiv  Detail & Related papers  (2025-11-04T16:31:44Z)
• Evaluating Large Language Models on Non-Code Software Engineering Tasks [4.381476817430934]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in code understanding and generation.<n>We present the first comprehensive benchmark, which we name Software Engineering Language Understanding' (SELU)<n>SELU covers classification, regression, Named Entity Recognition (NER) and Masked Language Modeling (MLM) targets, with data drawn from diverse sources.
  arXiv  Detail & Related papers  (2025-06-12T15:52:32Z)
• Codetations: Intelligent, Persistent Notes and UIs for Programs and Other Documents [0.85830154886823]
  We present Codetations, a system that helps developers contextualize documents with rich notes and tools.<n>Unlike previous approaches, notes in Codetations stay outside the document to prevent code clutter, attaching to spans in the document using a hybrid edit-tracking/LLM-based method.<n>Their content is dynamic, interactive, and synchronized with code changes.
  arXiv  Detail & Related papers  (2025-04-25T21:33:25Z)
• Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation [96.18720164390699]
  This paper explores the use of large language models (LLMs) for annotating document utility in training retrieval and retrieval-augmented generation (RAG) systems.<n>Our results show that LLM-generated annotations enhance out-of-domain retrieval performance and improve RAG outcomes compared to models trained solely on human annotations or downstream QA metrics.
  arXiv  Detail & Related papers  (2025-04-07T16:05:52Z)
• SAGEval: The frontiers of Satisfactory Agent based NLG Evaluation for reference-free open-ended text [0.848663031844483]
  This paper identifies the need to develop robust evaluation approaches for natural language generation, wherein references/ground labels doesn't exist or isn't amply available. We show that the critiquing Agent is able to rectify scores from LLM evaluators, thereby reducing the need for labeled data even for complex NLG evaluation scenarios.
  arXiv  Detail & Related papers  (2024-11-25T04:07:16Z)
• What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering [8.019873464066308]
  We introduce two metrics for classification tasks, namely sensitivity and consistency.<n> sensitivity measures changes of predictions across rephrasings of the prompt.<n>Instead, consistency measures how predictions vary across rephrasings for elements of the same class.
  arXiv  Detail & Related papers  (2024-06-18T06:59:24Z)
• PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion [96.47420221442397]
  We construct adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels. We test 3 closed-source and 4 open-source LLMs using a benchmark that incorporates robustness settings. We find that GPT-4 exhibits the highest performance and strong robustness in our benchmark.
  arXiv  Detail & Related papers  (2024-03-06T15:33:32Z)
• ThinkNote: Enhancing Knowledge Integration and Utilization of Large Language Models via Constructivist Cognition Modeling [55.21641515545307]
  Large Language Models (LLMs) have demonstrated strong performance across a wide range of NLP tasks.<n>They often exhibit suboptimal behaviors and inconsistencies when exposed to unfamiliar external information.<n>We propose ThinkNote, a novel framework that enhances the external knowledge utilization of LLMs.
  arXiv  Detail & Related papers  (2024-02-21T06:04:53Z)
• CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
  We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
  arXiv  Detail & Related papers  (2023-11-29T08:29:54Z)
• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation [94.59630161324013]
  We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline.
  arXiv  Detail & Related papers  (2023-10-24T08:56:49Z)
• Beyond the Chat: Executable and Verifiable Text-Editing with LLMs [87.84199761550634]
  Conversational interfaces powered by Large Language Models (LLMs) have recently become a popular way to obtain feedback during document editing. We present InkSync, an editing interface that suggests executable edits directly within the document being edited.
  arXiv  Detail & Related papers  (2023-09-27T00:56:17Z)
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models [73.86954509967416]
  Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks. This paper presents the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks.
  arXiv  Detail & Related papers  (2023-06-23T09:22:36Z)
• Automated Annotation with Generative AI Requires Validation [0.0]
  Generative large language models (LLMs) can be a powerful tool for augmenting text annotation procedures. We outline a workflow to harness the annotation potential of LLMs in a principled, efficient way. We find that LLM performance for text annotation is promising but highly contingent on both the dataset and the type of annotation task.
  arXiv  Detail & Related papers  (2023-05-31T20:50:45Z)
• EditEval: An Instruction-Based Benchmark for Text Improvements [73.5918084416016]
  This work presents EditEval: An instruction-based, benchmark and evaluation suite for automatic evaluation of editing capabilities. We evaluate several pre-trained models, which shows that InstructGPT and PEER perform the best, but that most baselines fall below the supervised SOTA. Our analysis shows that commonly used metrics for editing tasks do not always correlate well, and that optimization for prompts with the highest performance does not necessarily entail the strongest robustness to different models.
  arXiv  Detail & Related papers  (2022-09-27T12:26:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.