Towards Lean Research Inception: Assessing Practical Relevance of Formulated Research Problems

• URL: http://arxiv.org/abs/2506.12669v1
• Date: Sun, 15 Jun 2025 00:23:48 GMT
• Title: Towards Lean Research Inception: Assessing Practical Relevance of Formulated Research Problems
• Authors: Anrafel Fernandes Pereira, Marcos Kalinowski, Maria Teresa Baldassarre, Jürgen Börstler, Nauman bin Ali, Daniel Mendez,
• Abstract summary: Lack of practical relevance in many Software Engineering (SE) research contributions is often rooted in oversimplified views of industrial practice, weak industry connections, and poorly defined research problems.<n>We introduce the Lean Research Inception (LRI) framework, designed to support the formulation and assessment of practically relevant research problems in SE.<n>We describe its initial evaluation strategy conducted in a workshop with a network of SE researchers experienced in industry-academia collaboration.
• Score: 6.1925991198428365
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: [Context] The lack of practical relevance in many Software Engineering (SE) research contributions is often rooted in oversimplified views of industrial practice, weak industry connections, and poorly defined research problems. Clear criteria for evaluating SE research problems can help align their value, feasibility, and applicability with industrial needs. [Goal] In this paper, we introduce the Lean Research Inception (LRI) framework, designed to support the formulation and assessment of practically relevant research problems in SE. We describe its initial evaluation strategy conducted in a workshop with a network of SE researchers experienced in industry-academia collaboration and report the evaluation of its three assessment criteria (valuable, feasible, and applicable) regarding their importance in assessing practical relevance. [Method] We applied LRI retroactively to a published research paper, engaging workshop participants in discussing and assessing the research problem by applying the proposed criteria using a semantic differential scale. Participants provided feedback on the criteria's importance and completeness, drawn from their own experiences in industry-academia collaboration. [Results] The findings reveal an overall agreement on the importance of the three criteria - valuable (83.3%), feasible (76.2%), and applicable (73.8%) - for aligning research problems with industrial needs. Qualitative feedback suggested adjustments in terminology with a clearer distinction between feasible and applicable, and refinements for valuable by more clearly considering business value, ROI, and originality. [Conclusion] While LRI constitutes ongoing research and requires further evaluation, our results strengthen our confidence that the three criteria applied using the semantic differential scale can already help the community assess the practical relevance of SE research problems.

Related papers

• The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research [56.80927148740585]
  We address the challenges of scalability and rigor by flipping the dynamic and developing AI agents as research evaluators.<n>We use mechanistic interpretability research as a testbed, build standardized research output, and develop MechEvalAgent.<n>Our work demonstrates the potential of AI agents to transform research evaluation and pave the way for rigorous scientific practices.
  arXiv  Detail & Related papers  (2026-02-05T19:00:02Z)
• DeepSurvey-Bench: Evaluating Academic Value of Automatically Generated Scientific Survey [53.85391477976017]
  DeepSurvey-Bench is a novel benchmark designed to comprehensively evaluate the academic value of generated surveys.<n>We construct a reliable dataset with academic value annotations, and evaluate the deep academic value of the generated surveys.
  arXiv  Detail & Related papers  (2026-01-13T14:42:56Z)
• Aligning Academia with Industry: An Empirical Study of Industrial Needs and Academic Capabilities in AI-Driven Software Engineering [45.09204791294318]
  The rapid advancement of large language models (LLMs) is fundamentally reshaping software engineering (SE)<n>While top-tier SE venues continue to show sustained or emerging focus on areas like automated testing and program repair, the alignment of these academic advances with real industrial needs remains unclear.<n>This study aims to refocus academic attention on these important yet under-explored problems and to guide future SE research toward greater industrial impact.
  arXiv  Detail & Related papers  (2025-12-17T07:29:18Z)
• Towards AI Agents Supported Research Problem Formulation [3.6732711233211663]
  Poorly formulated research problems can compromise the practical relevance of Software Engineering studies.<n>This vision paper explores the use of artificial intelligence agents to support SE researchers during the early stage of a research project.
  arXiv  Detail & Related papers  (2025-12-14T14:44:27Z)
• Attributes to Support the Formulation of Practically Relevant Research Problems in Software Engineering [8.029394536536483]
  A well-formulated research problem is essential for achieving practical relevance in Software Engineering (SE)<n>Our goal is to introduce and evaluate seven attributes identified in the SE literature as relevant for formulating research problems.
  arXiv  Detail & Related papers  (2025-12-14T14:06:25Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• Point of Interest Recommendation: Pitfalls and Viable Solutions [44.68478552919453]
  Point of interest (POI) recommendation can play a pivotal role in enriching tourists' experiences.<n>POI recommendation is inherently high-stakes: users invest significant time, money, and effort to search, choose, and consume these suggested POIs.<n>Despite the numerous research works in the area, several fundamental issues remain unresolved.
  arXiv  Detail & Related papers  (2025-07-18T08:10:09Z)
• OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models [55.21589313404023]
  OpenReview is a continually evolving repository of research papers, peer reviews, author rebuttals, meta-reviews, and decision outcomes.<n>We highlight three promising areas in which OpenReview can uniquely contribute: enhancing the quality, scalability, and accountability of peer review processes; enabling meaningful, open-ended benchmarks rooted in genuine expert deliberation; and supporting alignment research through real-world interactions reflecting expert assessment, intentions, and scientific values.<n>We suggest the community collaboratively explore standardized benchmarks and usage guidelines around OpenReview, inviting broader dialogue on responsible data use, ethical considerations, and collective stewardship.
  arXiv  Detail & Related papers  (2025-05-24T09:07:13Z)
• A Call for Critically Rethinking and Reforming Data Analysis in Empirical Software Engineering [5.687882380471718]
  Concerns about the correct application of empirical methodologies have existed since the 2006 Dagstuhl seminar on Empirical Software Engineering.<n>We conducted a literature survey of 27,000 empirical studies, using LLMs to classify statistical methodologies as adequate or inadequate.<n>We selected 30 primary studies and held a workshop with 33 ESE experts to assess their ability to identify and resolve statistical issues.
  arXiv  Detail & Related papers  (2025-01-22T09:05:01Z)
• AAAR-1.0: Assessing AI's Potential to Assist Research [34.88341605349765]
  We introduce AAAR-1.0, a benchmark dataset designed to evaluate large language models (LLMs) performance in three fundamental, expertise-intensive research tasks.<n> AAAR-1.0 differs from prior benchmarks in two key ways: first, it is explicitly research-oriented, with tasks requiring deep domain expertise; second, it is researcher-oriented, mirroring the primary activities that researchers engage in on a daily basis.
  arXiv  Detail & Related papers  (2024-10-29T17:58:29Z)
• Iterative Utility Judgment Framework via LLMs Inspired by Relevance in Philosophy [66.95501113584541]
  We propose an Iterative utiliTy judgm fraEntMework (ITEM) to promote each step in Retrieval-Augmented Generation (RAG)<n>RAG's three core components -- relevance ranking derived from retrieval models, utility judgments, and answer generation -- align with Schutz's philosophical system of relevances.<n> Experimental results demonstrate significant improvements of ITEM in utility judgments, ranking, and answer generation upon representative baselines.
  arXiv  Detail & Related papers  (2024-06-17T07:52:42Z)
• Are we making progress in unlearning? Findings from the first NeurIPS unlearning competition [70.60872754129832]
  First NeurIPS competition on unlearning sought to stimulate the development of novel algorithms. Nearly 1,200 teams from across the world participated. We analyze top solutions and delve into discussions on benchmarking unlearning.
  arXiv  Detail & Related papers  (2024-06-13T12:58:00Z)
• Privacy Impact Assessments in the Wild: A Scoping Review [1.7677916783208343]
  Privacy Impact Assessments (PIAs) offer a systematic process for assessing the privacy impacts of a project or system. PIAs are heralded as one of the main approaches to privacy by design, supporting the early identification of threats and controls. There is still a significant need for more primary research on the topic, both qualitative and quantitative.
  arXiv  Detail & Related papers  (2024-02-17T05:07:10Z)
• Insights Towards Better Case Study Reporting in Software Engineering [0.0]
  This paper aims to share insights that can enhance the quality and impact of case study reporting. We emphasize the need to follow established guidelines, accurate classification, and detailed context descriptions in case studies. We aim to encourage researchers to adopt more rigorous and communicative strategies, ensuring that case studies are methodologically sound.
  arXiv  Detail & Related papers  (2024-02-13T12:29:26Z)
• Responsible AI Considerations in Text Summarization Research: A Review of Current Practices [89.85174013619883]
  We focus on text summarization, a common NLP task largely overlooked by the responsible AI community. We conduct a multi-round qualitative analysis of 333 summarization papers from the ACL Anthology published between 2020-2022. We focus on how, which, and when responsible AI issues are covered, which relevant stakeholders are considered, and mismatches between stated and realized research goals.
  arXiv  Detail & Related papers  (2023-11-18T15:35:36Z)
• A Survey on Interpretable Cross-modal Reasoning [64.37362731950843]
  Cross-modal reasoning (CMR) has emerged as a pivotal area with applications spanning from multimedia analysis to healthcare diagnostics. This survey delves into the realm of interpretable cross-modal reasoning (I-CMR) This survey presents a comprehensive overview of the typical methods with a three-level taxonomy for I-CMR.
  arXiv  Detail & Related papers  (2023-09-05T05:06:48Z)
• Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research [75.84463664853125]
  We provide a first attempt to quantify concerns regarding three topics, namely, environmental impact, equity, and impact on peer reviewing. We capture existing (dis)parities between different and within groups with respect to seniority, academia, and industry. We devise recommendations to mitigate found disparities, some of which already successfully implemented.
  arXiv  Detail & Related papers  (2023-06-29T12:44:53Z)
• Fairness in Recommender Systems: Research Landscape and Future Directions [119.67643184567623]
  We review the concepts and notions of fairness that were put forward in the area in the recent past. We present an overview of how research in this field is currently operationalized. Overall, our analysis of recent works points to certain research gaps.
  arXiv  Detail & Related papers  (2022-05-23T08:34:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.