LLM-based Models for Detecting Emerging Topics in Service Feedback

• URL: http://arxiv.org/abs/2606.26595v1
• Date: Thu, 25 Jun 2026 04:46:11 GMT
• Title: LLM-based Models for Detecting Emerging Topics in Service Feedback
• Abstract summary: This paper presents a novel methodology that integrates large language models (LLMs), statistical techniques, and human-AI collaboration to improve multilingual customer feedback analysis.<n>The primary objective is to detect emerging service quality topics that may also reveal potential inequities in service delivery.
• Score: 1.5484595752241122
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Enhancing the analysis of service feedback is essential for public sector organizations, particularly tax administrations, where trust and compliance depend on fair and effective service delivery. As feedback volumes grow, identifying emerging service quality issues and potential disparities across diverse populations becomes increasingly challenging. Traditional approaches often rely on manual review or static expert-defined indicators, limiting scalability and the ability to capture complex patterns in textual feedback. This paper presents a novel methodology that integrates large language models (LLMs), statistical techniques, and human-AI collaboration to improve multilingual customer feedback analysis. The primary objective is to detect emerging service quality topics that may also reveal potential inequities in service delivery. Our framework combines fine-tuned, quantized LLMs with expert oversight to produce accurate, computationally efficient, and context-aware analyses. The proposed approach was evaluated using similarity analysis and assessments from experienced tax officers, demonstrating stronger alignment with expert judgments than baseline models. By incorporating a human-in-the-loop framework, the methodology reduces LLM fabrication while improving the reliability and relevance of generated insights. The results demonstrate the practicality of combining LLMs with human expertise to support scalable, evidence-based decision-making in public sector organizations. This work contributes to the development of responsible AI systems that enhance service quality, responsiveness, fairness, and public trust through more effective analysis of multilingual customer feedback.

Related papers

• LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges [4.655159257282136]
  This survey offers a systems-level analysis of large language models (LLMs)-based scientific peer review.<n>We focus on two core evaluative functions: critique generation and score prediction.<n>We identify emerging risks, including prompt injection, data poisoning, retrieval vulnerabilities, and reward hacking.
  arXiv  Detail & Related papers  (2026-06-23T18:12:39Z)
• Leveraging Large Language Models for Trustworthiness Assessment of Web Applications [13.909850314037653]
  This study presents an empirical methodology to automate the trustworthiness assessment of web applications by leveraging Large Language Models (LLMs)<n>We propose an extension of a hierarchical Quality Model (QM) based on the Logic Score of Preference (LSP)<n> Experimental results indicate that excessive structural context can introduce noise, whereas rule-based instructional prompting improves assessment reliability.
  arXiv  Detail & Related papers  (2026-03-24T23:33:54Z)
• Towards Recommending Usability Improvements with Multimodal Large Language Models [40.77787659104315]
  Common evaluation methods, such as usability testing and inspection, are effective but resource-intensive and require expert involvement.<n>Recent advances in multimodal LLMs offer promising opportunities to automate usability evaluation processes.<n>Our findings indicate the potential of LLMs to enable faster and more cost-effective usability evaluation.
  arXiv  Detail & Related papers  (2025-08-22T07:38:37Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• Fine-Grained Bias Detection in LLM: Enhancing detection mechanisms for nuanced biases [0.0]
  This study presents a detection framework to identify nuanced biases in Large Language Models (LLMs)<n>The approach integrates contextual analysis, interpretability via attention mechanisms, and counterfactual data augmentation to capture hidden biases.<n>Results show improvements in detecting subtle biases compared to conventional methods.
  arXiv  Detail & Related papers  (2025-03-08T04:43:01Z)
• An Overview of Large Language Models for Statisticians [109.38601458831545]
  Large Language Models (LLMs) have emerged as transformative tools in artificial intelligence (AI)<n>This paper explores potential areas where statisticians can make important contributions to the development of LLMs.<n>We focus on issues such as uncertainty quantification, interpretability, fairness, privacy, watermarking and model adaptation.
  arXiv  Detail & Related papers  (2025-02-25T03:40:36Z)
• Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis [4.539569292151314]
  Large Language Models (LLMs) enable human-bot collaboration in Software Engineering (SE)<n>This study is to design and develop an LLM-based multi-agent system that synergizes human decision support with AI to automate various qualitative data analysis approaches.
  arXiv  Detail & Related papers  (2024-02-02T13:10:46Z)
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents [74.16170899755281]
  We introduce AgentBoard, a pioneering comprehensive benchmark and accompanied open-source evaluation framework tailored to analytical evaluation of LLM agents.<n>AgentBoard offers a fine-grained progress rate metric that captures incremental advancements as well as a comprehensive evaluation toolkit.<n>This not only sheds light on the capabilities and limitations of LLM agents but also propels the interpretability of their performance to the forefront.
  arXiv  Detail & Related papers  (2024-01-24T01:51:00Z)
• Leveraging Large Language Models for NLG Evaluation: Advances and Challenges [57.88520765782177]
  Large Language Models (LLMs) have opened new avenues for assessing generated content quality, e.g., coherence, creativity, and context relevance. We propose a coherent taxonomy for organizing existing LLM-based evaluation metrics, offering a structured framework to understand and compare these methods. By discussing unresolved challenges, including bias, robustness, domain-specificity, and unified evaluation, this paper seeks to offer insights to researchers and advocate for fairer and more advanced NLG evaluation techniques.
  arXiv  Detail & Related papers  (2024-01-13T15:59:09Z)
• MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation [60.65820977963331]
  We introduce a novel evaluation paradigm for Large Language Models (LLMs) This paradigm shifts the emphasis from result-oriented assessments, which often neglect the reasoning process, to a more comprehensive evaluation. By applying this paradigm in the GSM8K dataset, we have developed the MR-GSM8K benchmark.
  arXiv  Detail & Related papers  (2023-12-28T15:49:43Z)
• ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate [57.71597869337909]
  We build a multi-agent referee team called ChatEval to autonomously discuss and evaluate the quality of generated responses from different models. Our analysis shows that ChatEval transcends mere textual scoring, offering a human-mimicking evaluation process for reliable assessments.
  arXiv  Detail & Related papers  (2023-08-14T15:13:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.