On Systematically Building a Controlled Natural Language for Functional
Requirements
- URL: http://arxiv.org/abs/2005.01355v1
- Date: Mon, 4 May 2020 09:55:38 GMT
- Title: On Systematically Building a Controlled Natural Language for Functional
Requirements
- Authors: Alvaro Veizaga, Mauricio Alferez, Damiano Torre, Mehrdad Sabetzadeh,
Lionel Briand
- Abstract summary: Natural language (NL) is pervasive in software requirements specifications (SRSs)
Despite its popularity and widespread use, NL is highly prone to quality issues such as vagueness, ambiguity, and incompleteness.
Controlled natural languages (CNLs) have been proposed as a way to prevent quality problems in requirements documents.
- Score: 2.9676973500772887
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: [Context] Natural language (NL) is pervasive in software requirements
specifications (SRSs). However, despite its popularity and widespread use, NL
is highly prone to quality issues such as vagueness, ambiguity, and
incompleteness. Controlled natural languages (CNLs) have been proposed as a way
to prevent quality problems in requirements documents, while maintaining the
flexibility to write and communicate requirements in an intuitive and
universally understood manner. [Objective] In collaboration with an industrial
partner from the financial domain, we systematically develop and evaluate a
CNL, named Rimay, intended at helping analysts write functional requirements.
[Method] We rely on Grounded Theory for building Rimay and follow well-known
guidelines for conducting and reporting industrial case study research.
[Results] Our main contributions are: (1) a qualitative methodology to
systematically define a CNL for functional requirements; this methodology is
general and applicable to information systems beyond the financial domain, (2)
a CNL grammar to represent functional requirements; this grammar is derived
from our experience in the financial domain, but should be applicable, possibly
with adaptations, to other information-system domains, and (3) an empirical
evaluation of our CNL (Rimay) through an industrial case study. Our
contributions draw on 15 representative SRSs, collectively containing 3215 NL
requirements statements from the financial domain. [Conclusion] Our evaluation
shows that Rimay is expressive enough to capture, on average, 88% (405 out of
460) of the NL requirements statements in four previously unseen SRSs from the
financial domain.
Related papers
- CFinBench: A Comprehensive Chinese Financial Benchmark for Large Language Models [61.324062412648075]
CFinBench is an evaluation benchmark for assessing the financial knowledge of large language models (LLMs) under Chinese context.
It comprises 99,100 questions spanning 43 second-level categories with 3 question types: single-choice, multiple-choice and judgment.
The results show that GPT4 and some Chinese-oriented models lead the benchmark, with the highest average accuracy being 60.16%.
arXiv Detail & Related papers (2024-07-02T14:34:36Z) - SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal Behaviors [64.9938658716425]
Existing evaluations of large language models' (LLMs) ability to recognize and reject unsafe user requests face three limitations.
First, existing methods often use coarse-grained of unsafe topics, and are over-representing some fine-grained topics.
Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
Third, existing evaluations rely on large LLMs for evaluation, which can be expensive.
arXiv Detail & Related papers (2024-06-20T17:56:07Z) - Leveraging Large Language Models for NLG Evaluation: Advances and Challenges [57.88520765782177]
Large Language Models (LLMs) have opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance.
We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods.
By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this paper seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.
arXiv Detail & Related papers (2024-01-13T15:59:09Z) - Practical Guidelines for the Selection and Evaluation of Natural Language Processing Techniques in Requirements Engineering [8.779031107963942]
Natural language (NL) is now a cornerstone of requirements automation.
With so many different NLP solution strategies available, it can be challenging to choose the right strategy for a specific RE task.
In particular, we discuss how to choose among different strategies such as traditional NLP, feature-based machine learning, and language-model-based methods.
arXiv Detail & Related papers (2024-01-03T02:24:35Z) - Status Quo and Problems of Requirements Engineering for Machine
Learning: Results from an International Survey [7.164324501049983]
Requirements Engineering (RE) can help address many problems when engineering Machine Learning-enabled systems.
We conducted a survey to gather practitioner insights into the status quo and problems of RE in ML-enabled systems.
We found significant differences in RE practices within ML projects.
arXiv Detail & Related papers (2023-10-10T15:53:50Z) - Situated Natural Language Explanations [54.083715161895036]
Natural language explanations (NLEs) are among the most accessible tools for explaining decisions to humans.
Existing NLE research perspectives do not take the audience into account.
Situated NLE provides a perspective and facilitates further research on the generation and evaluation of explanations.
arXiv Detail & Related papers (2023-08-27T14:14:28Z) - Automated Smell Detection and Recommendation in Natural Language
Requirements [8.672583050502496]
Paska is a tool that takes as input any natural language (NL) requirements.
It automatically detects quality problems as smells in the requirements, and offers recommendations to improve their quality.
arXiv Detail & Related papers (2023-05-11T19:01:25Z) - IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named
Entity Recognition using Knowledge Bases [53.054598423181844]
We present a novel NER cascade approach comprising three steps.
We empirically demonstrate the significance of external knowledge bases in accurately classifying fine-grained and emerging entities.
Our system exhibits robust performance in the MultiCoNER2 shared task, even in the low-resource language setting.
arXiv Detail & Related papers (2023-04-20T20:30:34Z) - Requirement Formalisation using Natural Language Processing and Machine
Learning: A Systematic Review [11.292853646607888]
We conducted a systematic literature review to outline the current state-of-the-art of NLP and ML techniques in Requirement Engineering.
We found that NLP approaches are the most common NLP techniques used for automatic RF, primary operating on structured and semi-structured data.
This study also revealed that Deep Learning (DL) technique are not widely used, instead classical ML techniques are predominant in the surveyed studies.
arXiv Detail & Related papers (2023-03-18T17:36:21Z) - The Use of NLP-Based Text Representation Techniques to Support
Requirement Engineering Tasks: A Systematic Mapping Review [1.5469452301122177]
The research direction has changed from the use of lexical and syntactic features to the use of advanced embedding techniques.
We identify four gaps in the existing literature, why they matter, and how future research can begin to address them.
arXiv Detail & Related papers (2022-05-17T02:47:26Z) - Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and
Their Implications [85.24952708195582]
This study examines the goals, community practices, assumptions, and constraints that shape NLG evaluations.
We examine their implications and how they embody ethical considerations.
arXiv Detail & Related papers (2022-05-13T18:00:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.