Improving Customer Service with Automatic Topic Detection in User Emails

• URL: http://arxiv.org/abs/2502.19115v3
• Date: Fri, 06 Jun 2025 12:12:39 GMT
• Title: Improving Customer Service with Automatic Topic Detection in User Emails
• Abstract summary: This study introduces a novel natural language processing pipeline that enhances customer service efficiency at Telekom Srbija.<n>Central to the pipeline is BERTopic, a modular framework that allows unsupervised topic modeling.<n>The system now operates in the company's production environment, streamlining customer service operations through automated email classification.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study introduces a novel natural language processing pipeline that enhances customer service efficiency at Telekom Srbija, a leading Serbian telecommunications company, through automated email topic detection and labeling. Central to the pipeline is BERTopic, a modular framework that allows unsupervised topic modeling. After a series of preprocessing and postprocessing steps, we assign one of 12 topics and several additional labels to incoming emails, allowing customer service to filter and access them through a custom-made application. While applied to Serbian, the methodology is conceptually language-agnostic and can be readily adapted to other languages, particularly those that are low-resourced and morphologically rich. The system performance was evaluated by assessing the speed and correctness of the automatically assigned topics, with a weighted average processing time of 0.041 seconds per email and a weighted average F1 score of 0.96. The system now operates in the company's production environment, streamlining customer service operations through automated email classification.

Related papers

• Content-Based Smart E-Mail Dispatcher Using Large Language Models [0.26097841018267615]
  The main objective of this paper is to explore an alternative mechanism that is to automate the task of dispatching emails based on their contents to the respective WhatsApp groups of students of various semesters of programs in an engineering college.<n>The dispatcher system is built using agents querying large language models (LLMs) to enable it to analyze the contents of emails and route them to the relevant groups of students for their information and consumption.<n>The proposed system does not rely on labelled datasets and provides several benefits, including enhanced productivity and a reduction in the cognitive load associated with reading emails.
  arXiv  Detail & Related papers  (2026-06-25T04:34:03Z)
• Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios [63.167237308464536]
  Speech translation (ST) is increasingly adopted in user applications, yet its evaluation focuses on decontextualized testbeds and holistic quality.<n>We introduce Ouvia, an evaluation framework for measuring user-perceived usability of speech translation outputs in real-world settings.
  arXiv  Detail & Related papers  (2026-06-04T13:52:21Z)
• ClawEnvKit: Automatic Environment Generation for Claw-Like Agents [85.29126619772153]
  We introduce ClawEnvKit, an autonomous generation pipeline capable of generating verified environments on demand.<n>ClawEnvKit comprises three modules: (1) a pipeline that extracts structured generation parameters from natural language input; (2) a generator that produces the task specification, tool interface, and scoring configuration; and (3) a validator that enforces feasibility, diversity, structural validity, and internal consistency.<n>We construct Auto-ClawEval, the first large-scale benchmark for claw-like agents, comprising 1,040 environments across 24 categories.
  arXiv  Detail & Related papers  (2026-04-20T17:36:49Z)
• AI Appeals Processor: A Deep Learning Approach to Automated Classification of Citizen Appeals in Government Services [0.0]
  Government agencies worldwide face growing volumes of citizen appeals.<n>Traditional manual processing averages 20 minutes per appeal with only 67% classification accuracy.<n>This paper presents AI Appeals Processor, a microservice-based system that integrates natural language processing and deep learning techniques.
  arXiv  Detail & Related papers  (2026-04-04T10:03:53Z)
• Tunable Automation in Automated Program Verification [42.02726718338287]
  SMT-based verification tools face a tension between automation and performance when dealing with quantifier instantiation.<n>We present a mechanism that enables fine-grained control over the availability of quantified facts in verification contexts.<n>We implement our techniques in Verus, a Rust-based verification tool, and evaluate them on multiple openly availables.
  arXiv  Detail & Related papers  (2025-12-03T16:27:01Z)
• AutoMalDesc: Large-Scale Script Analysis for Cyber Threat Research [81.04845910798387]
  Generating natural language explanations for threat detections remains an open problem in cybersecurity research.<n>We present AutoMalDesc, an automated static analysis summarization framework that operates independently at scale.<n>We publish our complete dataset of more than 100K script samples, including annotated seed (0.9K) datasets, along with our methodology and evaluation framework.
  arXiv  Detail & Related papers  (2025-11-17T13:05:25Z)
• Enhancing Customer Service Chatbots with Context-Aware NLU through Selective Attention and Multi-task Learning [0.0]
  We introduce a context-aware NLU model for predicting customer intent.<n>A novel selective attention module is used to extract relevant context features.<n>We have deployed our model to production for Walmart's customer care domain.
  arXiv  Detail & Related papers  (2025-06-02T15:24:28Z)
• Beyond General Prompts: Automated Prompt Refinement using Contrastive Class Alignment Scores for Disambiguating Objects in Vision-Language Models [0.0]
  We introduce a method for automated prompt refinement using a novel metric called the Contrastive Class Alignment Score (CCAS)<n>Our method generates diverse prompt candidates via a large language model and filters them through CCAS, computed using prompt embeddings from a sentence transformer.<n>We evaluate our approach on challenging object categories, demonstrating that our automatic selection of high-precision prompts improves object detection accuracy without the need for model training or labeled data.
  arXiv  Detail & Related papers  (2025-05-14T04:43:36Z)
• ScalerEval: Automated and Consistent Evaluation Testbed for Auto-scalers in Microservices [7.4883371370599585]
  ScalerEval is an end-to-end automated and consistent testbed for auto-scalers in researchers. It integrates essential fundamental interfaces for implementation of auto-scalers and further orchestrates a one-click evaluation for researchers.
  arXiv  Detail & Related papers  (2025-04-11T07:18:53Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents [65.36060818857109]
  We present a novel framework for extracting and evaluating dialog from historical interactions.<n>Our extraction process consists of two key stages: (1) a retrieval step to select relevant conversations based on key procedural elements, and (2) a structured workflow generation process using a question-answer-based chain-of-thought (QA-CoT) prompting.
  arXiv  Detail & Related papers  (2025-02-24T16:55:15Z)
• Text-Based Detection of On-Hold Scripts in Contact Center Calls [0.6138671548064356]
  Average hold time is a concern for call centers because it affects customer satisfaction. This study presents a natural language processing model that detects on-hold phrases in customer service calls transcribed by automatic speech recognition technology.
  arXiv  Detail & Related papers  (2024-07-13T11:11:41Z)
• Zero-shot prompt-based classification: topic labeling in times of foundation models in German Tweets [1.734165485480267]
  We propose a new tool for automatically annotating text using written guidelines without providing training samples. Our results show that the prompt-based approach is comparable with the fine-tuned BERT but without any annotated training data. Our findings emphasize the ongoing paradigm shift in the NLP landscape, i.e., the unification of downstream tasks and elimination of the need for pre-labeled training data.
  arXiv  Detail & Related papers  (2024-06-26T10:44:02Z)
• Issue Report Validation in an Industrial Context [1.993607565985189]
  We work on 1,200 randomly selected issue reports in banking domain, written in Turkish. We manually label these reports for validity, and extract the relevant patterns indicating that they are invalid. Using the proposed feature extractors, we utilize a machine learning based approach to predict the issue reports' validity, performing a 0.77 F1-score.
  arXiv  Detail & Related papers  (2023-11-29T14:24:13Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)
• Overview of Robust and Multilingual Automatic Evaluation Metrics for Open-Domain Dialogue Systems at DSTC 11 Track 4 [51.142614461563184]
  This track in the 11th Dialogue System Technology Challenge (DSTC11) is part of the ongoing effort to promote robust and multilingual automatic evaluation metrics. This article describes the datasets and baselines provided to participants and discusses the submission and result details of the two proposed subtasks.
  arXiv  Detail & Related papers  (2023-06-22T10:50:23Z)
• eTOP: Early Termination of Pipelines for Faster Training of AutoML Systems [12.933957727351666]
  Finding the right AI/ML model is a complex and costly process. We propose eTOP Framework which works on top of any AutoML system.
  arXiv  Detail & Related papers  (2023-04-17T20:22:30Z)
• Intent Recognition in Conversational Recommender Systems [0.0]
  We introduce a pipeline to contextualize the input utterances in conversations. We then take the next step towards leveraging reverse feature engineering to link the contextualized input and learning model to support intent recognition.
  arXiv  Detail & Related papers  (2022-12-06T11:02:42Z)
• A pipeline and comparative study of 12 machine learning models for text classification [0.0]
  Text-based communication is highly favoured as a communication method, especially in business environments. Many machine learning methods for text classification have been proposed and incorporated into the services of most email providers. However, optimising text classification algorithms and finding the right tradeoff on their aggressiveness is still a major research problem.
  arXiv  Detail & Related papers  (2022-04-04T23:51:22Z)
• What is wrong with you?: Leveraging User Sentiment for Automatic Dialog Evaluation [73.03318027164605]
  We propose to use information that can be automatically extracted from the next user utterance as a proxy to measure the quality of the previous system response. Our model generalizes across both spoken and written open-domain dialog corpora collected from real and paid users.
  arXiv  Detail & Related papers  (2022-03-25T22:09:52Z)
• A Flexible Clustering Pipeline for Mining Text Intentions [6.599344783327053]
  We create a flexible and scalable clustering pipeline within the Verint Intent Manager. It integrates the fine-tuning of language models, a high performing k-NN library and community detection techniques. As deployed in the VIM application, this clustering pipeline produces high quality results.
  arXiv  Detail & Related papers  (2022-02-01T22:54:18Z)
• A Natural Language Processing and Deep Learning based Model for Automated Vehicle Diagnostics using Free-Text Customer Service Reports [3.970010025578998]
  We demonstrate a machine learning pipeline to improve automated vehicle diagnostics. First, Natural Language Processing (NLP) is used to automate the extraction of crucial information from free-text failure reports. Deep learning algorithms are employed to validate service requests and filter vague or misleading claims.
  arXiv  Detail & Related papers  (2021-11-29T21:41:34Z)
• Machine Translation Customization via Automatic Training Data Selection from the Web [97.98885151955467]
  We describe an approach for customizing machine translation systems on specific domains. We select data similar to the target customer data to train neural translation models. Finally, we train MT models on our automatically selected data, obtaining a system specialized to the target domain.
  arXiv  Detail & Related papers  (2021-02-20T03:29:41Z)
• Timely Communication in Federated Learning [65.1253801733098]
  We consider a global learning framework in which a parameter server (PS) trains a global model by using $n$ clients without actually storing the client data centrally at a cloud server. Under the proposed scheme, at each iteration, the PS waits for $m$ available clients and sends them the current model. We find the average age of information experienced by each client and numerically characterize the age-optimal $m$ and $k$ values for a given $n$.
  arXiv  Detail & Related papers  (2020-12-31T18:52:08Z)
• Turn-level Dialog Evaluation with Dialog-level Weak Signals for Bot-Human Hybrid Customer Service Systems [0.0]
  We developed a machine learning approach that quantifies multiple aspects of the success or values in Customer Service contacts, at anytime during the interaction. We show how it improves Amazon customer service quality in several applications.
  arXiv  Detail & Related papers  (2020-10-25T19:36:23Z)
• Learning with Weak Supervision for Email Intent Detection [56.71599262462638]
  We propose to leverage user actions as a source of weak supervision to detect intents in emails. We develop an end-to-end robust deep neural network model for email intent identification.
  arXiv  Detail & Related papers  (2020-05-26T23:41:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.