CMER: A Context-Aware Approach for Mining Ethical Concern-related App Reviews

• URL: http://arxiv.org/abs/2507.09049v2
• Date: Sun, 20 Jul 2025 04:27:14 GMT
• Title: CMER: A Context-Aware Approach for Mining Ethical Concern-related App Reviews
• Authors: Aakash Sorathiya, Gouri Ginde,
• Abstract summary: This study proposes CMER (A underlineContext-Aware Approach for underlineEthical Concern-related App underlineReviews) to extract ethical concern-related app reviews at scale.<n>CMER combines Natural Language Inference (NLI) and a decoder-only (LLaMA-like) Large Language Model (LLM)<n>We evaluated the validity of CMER by mining privacy and security-related reviews (PSRs) from the dataset of more than 382K app reviews of mobile investment apps.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the increasing proliferation of mobile applications in our daily lives, the concerns surrounding ethics have surged significantly. Users communicate their feedback in app reviews, frequently emphasizing ethical concerns, such as privacy and security. Incorporating these reviews has proved to be useful for many areas of software engineering (e.g., requirement engineering, testing, etc.). However, app reviews related to ethical concerns generally use domain-specific language and are typically overshadowed by more generic categories of user feedback, such as app reliability and usability. Thus, making automated extraction a challenging and time-consuming effort. This study proposes CMER (A \underline{C}ontext-Aware Approach for \underline{M}ining \underline{E}thical Concern-related App \underline{R}eviews), a novel approach that combines Natural Language Inference (NLI) and a decoder-only (LLaMA-like) Large Language Model (LLM) to extract ethical concern-related app reviews at scale. In CMER, NLI provides domain-specific context awareness by using domain-specific hypotheses, and the Llama-like LLM eliminates the need for labeled data in the classification task. We evaluated the validity of CMER by mining privacy and security-related reviews (PSRs) from the dataset of more than 382K app reviews of mobile investment apps. First, we evaluated four NLI models and compared the results of domain-specific hypotheses with generic hypotheses. Next, we evaluated three LLMs for the classification task. Finally, we combined the best NLI and LLM models (CMER) and extracted 2,178 additional PSRs overlooked by the previous study using a keyword-based approach, thus demonstrating the effectiveness of CMER. These reviews can be further refined into actionable requirement artifacts.

Related papers

• Can LLMs Generate Tabular Summaries of Science Papers? Rethinking the Evaluation Protocol [83.90769864167301]
  Literature review tables are essential for summarizing and comparing collections of scientific papers.<n>We explore the task of generating tables that best fulfill a user's informational needs given a collection of scientific papers.<n>Our contributions focus on three key challenges encountered in real-world use: (i) User prompts are often under-specified; (ii) Retrieved candidate papers frequently contain irrelevant content; and (iii) Task evaluation should move beyond shallow text similarity techniques.
  arXiv  Detail & Related papers  (2025-04-14T14:52:28Z)
• Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics [1.3707925738322797]
  We propose a new metric called as Leniency, which quantifies evaluation strictness relative to expert assessment.<n>Our comprehensive analysis demonstrates that emphquestion-specific rubrics significantly enhance logical assessment of code in educational settings.
  arXiv  Detail & Related papers  (2025-03-31T11:59:43Z)
• The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility? [54.18519360412294]
  Large Language Models (LLMs) must balance between rejecting harmful requests for safety and accommodating legitimate ones for utility.<n>This paper presents a Direct Preference Optimization (DPO) based alignment framework that achieves better overall performance.<n>We analyze experimental results obtained from testing DeepSeek-R1 on our benchmark and reveal the critical ethical concerns raised by this highly acclaimed model.
  arXiv  Detail & Related papers  (2025-01-20T06:35:01Z)
• SAGEval: The frontiers of Satisfactory Agent based NLG Evaluation for reference-free open-ended text [0.848663031844483]
  This paper identifies the need to develop robust evaluation approaches for natural language generation, wherein references/ground labels doesn't exist or isn't amply available. We show that the critiquing Agent is able to rectify scores from LLM evaluators, thereby reducing the need for labeled data even for complex NLG evaluation scenarios.
  arXiv  Detail & Related papers  (2024-11-25T04:07:16Z)
• Beyond Keywords: A Context-based Hybrid Approach to Mining Ethical Concern-related App Reviews [0.0]
  App reviews related to ethical concerns generally use domain-specific language and are expressed using a more varied vocabulary. This study proposes a novel Natural Language Processing (NLI) based approach that combines Natural Language Inference (NLI) and a decoder-only (LLaMA-like) Large Language Model (LLM) to extract ethical concern-related app reviews at scale.
  arXiv  Detail & Related papers  (2024-11-11T22:08:48Z)
• Towards Extracting Ethical Concerns-related Software Requirements from App Reviews [0.0]
  This study analyzes app reviews of the Uber mobile application (a popular taxi/ride app) We propose a novel approach that leverages a knowledge graph (KG) model to extract software requirements from app reviews. Our framework consists of three main components: developing an ontology with relevant entities and relations, extracting key entities from app reviews, and creating connections between them.
  arXiv  Detail & Related papers  (2024-07-19T04:50:32Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• LEARN: Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application [54.984348122105516]
  Llm-driven knowlEdge Adaptive RecommeNdation (LEARN) framework synergizes open-world knowledge with collaborative knowledge.<n>We propose an Llm-driven knowlEdge Adaptive RecommeNdation (LEARN) framework that synergizes open-world knowledge with collaborative knowledge.
  arXiv  Detail & Related papers  (2024-05-07T04:00:30Z)
• Software Vulnerability and Functionality Assessment using LLMs [0.8057006406834466]
  We investigate whether Large Language Models (LLMs) can aid with code reviews. Our investigation focuses on two tasks that we argue are fundamental to good reviews.
  arXiv  Detail & Related papers  (2024-03-13T11:29:13Z)
• Fake Alignment: Are LLMs Really Aligned Well? [91.26543768665778]
  This study investigates the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization.
  arXiv  Detail & Related papers  (2023-11-10T08:01:23Z)
• Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation [52.62492168507781]
  We propose a novel benchmark called Fairness of Recommendation via LLM (FaiRLLM) This benchmark comprises carefully crafted metrics and a dataset that accounts for eight sensitive attributes. By utilizing our FaiRLLM benchmark, we conducted an evaluation of ChatGPT and discovered that it still exhibits unfairness to some sensitive attributes when generating recommendations.
  arXiv  Detail & Related papers  (2023-05-12T16:54:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.