GEMS: Generative Expert Metric System through Iterative Prompt Priming

• URL: http://arxiv.org/abs/2410.00880v1
• Date: Tue, 1 Oct 2024 17:14:54 GMT
• Title: GEMS: Generative Expert Metric System through Iterative Prompt Priming
• Authors: Ti-Chung Cheng, Carmen Badea, Christian Bird, Thomas Zimmermann, Robert DeLine, Nicole Forsgren, Denae Ford,
• Abstract summary: Non-experts can find it unintuitive to create effective measures or transform theories into context-specific metrics. This technical report addresses this challenge by examining software communities within large software corporations. We propose a prompt-engineering framework inspired by neural activities, demonstrating that generative models can extract and summarize theories.
• Score: 18.0413505095456
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Across domains, metrics and measurements are fundamental to identifying challenges, informing decisions, and resolving conflicts. Despite the abundance of data available in this information age, not only can it be challenging for a single expert to work across multi-disciplinary data, but non-experts can also find it unintuitive to create effective measures or transform theories into context-specific metrics that are chosen appropriately. This technical report addresses this challenge by examining software communities within large software corporations, where different measures are used as proxies to locate counterparts within the organization to transfer tacit knowledge. We propose a prompt-engineering framework inspired by neural activities, demonstrating that generative models can extract and summarize theories and perform basic reasoning, thereby transforming concepts into context-aware metrics to support software communities given software repository data. While this research zoomed in on software communities, we believe the framework's applicability extends across various fields, showcasing expert-theory-inspired metrics that aid in triaging complex challenges.

Related papers

• GIVE: Structured Reasoning with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning framework that integrates the parametric and non-parametric memories. Our method facilitates a more logical and step-wise reasoning approach akin to experts' problem-solving, rather than gold answer retrieval.
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• DISCOVER: A Data-driven Interactive System for Comprehensive Observation, Visualization, and ExploRation of Human Behaviour [6.716560115378451]
  We introduce a modular, flexible, yet user-friendly software framework specifically developed to streamline computational-driven data exploration for human behavior analysis. Our primary objective is to democratize access to advanced computational methodologies, thereby enabling researchers across disciplines to engage in detailed behavioral analysis without the need for extensive technical proficiency.
  arXiv  Detail & Related papers  (2024-07-18T11:28:52Z)
• Navigating Expertise in Configurable Software Systems through the Maze of Variability [0.0]
  This research study investigates the distribution of development efforts in CSS. It also examines the engagement of designated experts with variable code in their assigned files.
  arXiv  Detail & Related papers  (2024-01-19T14:03:33Z)
• The Transformative Influence of Large Language Models on Software Development [0.0]
  Generalized Large Language Models (LLMs) have found their way into diverse domains. With LLMs increasingly serving as AI Pair Programming Assistants, it also presents critical challenges and open problems. Preliminary findings underscore pressing concerns about data privacy, bias, and misinformation. We identify 12 open problems that we have identified through our survey, covering these various domains.
  arXiv  Detail & Related papers  (2023-11-28T02:18:54Z)
• Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework [51.44863255495668]
  Multimodal reasoning is a critical component in the pursuit of artificial intelligence systems that exhibit human-like intelligence. We present Multi-Modal Reasoning(COCO-MMR) dataset, a novel dataset that encompasses an extensive collection of open-ended questions. We propose innovative techniques, including multi-hop cross-modal attention and sentence-level contrastive learning, to enhance the image and text encoders.
  arXiv  Detail & Related papers  (2023-07-24T08:58:25Z)
• Understanding metric-related pitfalls in image analysis validation [59.15220116166561]
  This work provides the first comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy.
  arXiv  Detail & Related papers  (2023-02-03T14:57:40Z)
• Regressing Relative Fine-Grained Change for Sub-Groups in Unreliable Heterogeneous Data Through Deep Multi-Task Metric Learning [0.5999777817331317]
  We investigate how techniques in multi-task metric learning can be applied for theregression of fine-grained change in real data. The techniques investigated are specifically tailored for handling heterogeneous data sources.
  arXiv  Detail & Related papers  (2022-08-11T12:57:11Z)
• Metrics reloaded: Recommendations for image analysis validation [59.60445111432934]
  Metrics Reloaded is a comprehensive framework guiding researchers in the problem-aware selection of metrics. The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint. Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics.
  arXiv  Detail & Related papers  (2022-06-03T15:56:51Z)
• GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning [172.36214872466707]
  We focus on solving geometric problems, which requires a comprehensive understanding of textual descriptions, visual diagrams, and theorem knowledge. We propose a Geometric Question Answering dataset GeoQA, containing 5,010 geometric problems with corresponding annotated programs.
  arXiv  Detail & Related papers  (2021-05-30T12:34:17Z)
• Understanding the Usability Challenges of Machine Learning In High-Stakes Decision Making [67.72855777115772]
  Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts -- who often have no expertise in ML or data science -- are asked to use ML predictions to make high-stakes decisions. We investigate the ML usability challenges present in the domain of child welfare screening through a series of collaborations with child welfare screeners.
  arXiv  Detail & Related papers  (2021-03-02T22:50:45Z)
• Synergizing Domain Expertise with Self-Awareness in Software Systems: A Patternized Architecture Guideline [11.155059219430207]
  This paper highlights the importance of synergizing domain expertise and the self-awareness to enable better self-adaptation in software systems. We present a holistic framework of notions, enriched patterns and methodology, dubbed DBASES, that offers a principled guideline for the engineers.
  arXiv  Detail & Related papers  (2020-01-20T12:17:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.