Attributes to Support the Formulation of Practically Relevant Research Problems in Software Engineering

• URL: http://arxiv.org/abs/2512.12699v1
• Date: Sun, 14 Dec 2025 14:06:25 GMT
• Title: Attributes to Support the Formulation of Practically Relevant Research Problems in Software Engineering
• Authors: Anrafel Fernandes Pereira, Maria Teresa Baldassarre, Daniel Mendez, Jürgen Börstler, Nauman bin Ali, Rahul Mohanani, Darja Smite, Stefan Biffl, Rogardt Heldal, Davide Falessi, Daniel Graziotin, Marcos Kalinowski,
• Abstract summary: A well-formulated research problem is essential for achieving practical relevance in Software Engineering (SE)<n>Our goal is to introduce and evaluate seven attributes identified in the SE literature as relevant for formulating research problems.
• Score: 8.029394536536483
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: [Background] A well-formulated research problem is essential for achieving practical relevance in Software Engineering (SE), yet there is a lack of structured guidance in this early phase. [Aims] Our goal is to introduce and evaluate seven attributes identified in the SE literature as relevant for formulating research problems (practical problem, context, implications/impacts, practitioners, evidence, objective, and research questions) in terms of their perceived importance and completeness, and learn how they can be applied. [Method] We conducted a workshop with 42 senior SE researchers during the ISERN 2024 meeting. The seven attributes were presented using a Problem Vision board filled with a research example. Participants discussed attributes in groups, shared written feedback, and individually completed a survey assessing their importance, completeness, and suggestions for improvement. [Results] The findings confirm the importance of the seven attributes in the formulation of industry-oriented research problems. Qualitative feedback illustrated how they can be applied in practice and revealed suggestions to refine them, such as incorporating financial criteria (e.g., ROI) into implications/impacts and addressing feasibility and constraints under evidence. [Conclusion] The results reaffirm the importance of the seven attributes in supporting a reflective and context-aware problem formulation. Adapting their use to specific research contexts can help to improve the alignment between academic research and industry needs.

Related papers

• The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research [56.80927148740585]
  We address the challenges of scalability and rigor by flipping the dynamic and developing AI agents as research evaluators.<n>We use mechanistic interpretability research as a testbed, build standardized research output, and develop MechEvalAgent.<n>Our work demonstrates the potential of AI agents to transform research evaluation and pave the way for rigorous scientific practices.
  arXiv  Detail & Related papers  (2026-02-05T19:00:02Z)
• Towards AI Agents Supported Research Problem Formulation [3.6732711233211663]
  Poorly formulated research problems can compromise the practical relevance of Software Engineering studies.<n>This vision paper explores the use of artificial intelligence agents to support SE researchers during the early stage of a research project.
  arXiv  Detail & Related papers  (2025-12-14T14:44:27Z)
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks [103.53142193393931]
  evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment.<n>We conduct a systematic review of 445 benchmarks from leading conferences in natural language processing and machine learning.<n>We find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims.
  arXiv  Detail & Related papers  (2025-11-03T17:39:40Z)
• QUINTA: Reflexive Sensibility For Responsible AI Research and Data-Driven Processes [2.504366738288215]
  This paper presents a comprehensive framework grounded in critical reflexivity as intersectional praxis.<n>The framework centers researcher reflexivity to call attention to the AI researchers' power in creating and analyzing AI/DS artifacts through data-centric approaches.
  arXiv  Detail & Related papers  (2025-09-19T18:40:30Z)
• Towards Lean Research Inception: Assessing Practical Relevance of Formulated Research Problems [6.1925991198428365]
  Lack of practical relevance in many Software Engineering (SE) research contributions is often rooted in oversimplified views of industrial practice, weak industry connections, and poorly defined research problems.<n>We introduce the Lean Research Inception (LRI) framework, designed to support the formulation and assessment of practically relevant research problems in SE.<n>We describe its initial evaluation strategy conducted in a workshop with a network of SE researchers experienced in industry-academia collaboration.
  arXiv  Detail & Related papers  (2025-06-15T00:23:48Z)
• The AI Imperative: Scaling High-Quality Peer Review in Machine Learning [49.87236114682497]
  We argue that AI-assisted peer review must become an urgent research and infrastructure priority.<n>We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making.
  arXiv  Detail & Related papers  (2025-06-09T18:37:14Z)
• Bridging the Gap: Integrating Ethics and Environmental Sustainability in AI Research and Practice [57.94036023167952]
  We argue that the efforts aiming to study AI's ethical ramifications should be made in tandem with those evaluating its impacts on the environment.<n>We propose best practices to better integrate AI ethics and sustainability in AI research and practice.
  arXiv  Detail & Related papers  (2025-04-01T13:53:11Z)
• A Comparative Study of Android Performance Issues in Real-world Applications and Literature [16.259447959343067]
  We conduct a large-scale comparative study of Android performance issues in real-world applications and literature.<n>Our results show a substantial divergence exists in the primary performance concerns of researchers, developers, and users.<n>It is crucial for our community to intensify efforts to bridge these gaps and achieve comprehensive detection and resolution of performance issues.
  arXiv  Detail & Related papers  (2024-07-06T14:43:40Z)
• Iterative Utility Judgment Framework via LLMs Inspired by Relevance in Philosophy [66.95501113584541]
  We propose an Iterative utiliTy judgm fraEntMework (ITEM) to promote each step in Retrieval-Augmented Generation (RAG)<n>RAG's three core components -- relevance ranking derived from retrieval models, utility judgments, and answer generation -- align with Schutz's philosophical system of relevances.<n> Experimental results demonstrate significant improvements of ITEM in utility judgments, ranking, and answer generation upon representative baselines.
  arXiv  Detail & Related papers  (2024-06-17T07:52:42Z)
• Towards leveraging LLMs for Conditional QA [1.9649272351760063]
  This study delves into the capabilities and limitations of Large Language Models (LLMs) in the challenging domain of conditional question-answering. Our findings reveal that fine-tuned LLMs can surpass the state-of-the-art (SOTA) performance in some cases, even without fully encoding all input context. These models encounter challenges in extractive question answering, where they lag behind the SOTA by over 10 points, and in mitigating the risk of injecting false information.
  arXiv  Detail & Related papers  (2023-12-02T14:02:52Z)
• Responsible AI Considerations in Text Summarization Research: A Review of Current Practices [89.85174013619883]
  We focus on text summarization, a common NLP task largely overlooked by the responsible AI community. We conduct a multi-round qualitative analysis of 333 summarization papers from the ACL Anthology published between 2020-2022. We focus on how, which, and when responsible AI issues are covered, which relevant stakeholders are considered, and mismatches between stated and realized research goals.
  arXiv  Detail & Related papers  (2023-11-18T15:35:36Z)
• Fairness in Recommender Systems: Research Landscape and Future Directions [119.67643184567623]
  We review the concepts and notions of fairness that were put forward in the area in the recent past. We present an overview of how research in this field is currently operationalized. Overall, our analysis of recent works points to certain research gaps.
  arXiv  Detail & Related papers  (2022-05-23T08:34:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.