MetRoBERTa: Leveraging Traditional Customer Relationship Management Data
to Develop a Transit-Topic-Aware Language Model
- URL: http://arxiv.org/abs/2308.05012v1
- Date: Wed, 9 Aug 2023 15:11:37 GMT
- Title: MetRoBERTa: Leveraging Traditional Customer Relationship Management Data
to Develop a Transit-Topic-Aware Language Model
- Authors: Michael Leong, Awad Abdelhalim, Jude Ha, Dianne Patterson, Gabriel L.
Pincus, Anthony B. Harris, Michael Eichler, Jinhua Zhao
- Abstract summary: We propose a transit-topic-aware large language model (LLM) capable of classifying open-ended text feedback to relevant transit-specific topics.
First, we utilize semi-supervised learning to engineer a training dataset of 11 broad transit topics detected in a corpus of 6 years of customer feedback.
We then use this dataset to train and thoroughly evaluate a language model based on the RoBERTa architecture.
- Score: 3.3421154214189284
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Transit riders' feedback provided in ridership surveys, customer relationship
management (CRM) channels, and in more recent times, through social media is
key for transit agencies to better gauge the efficacy of their services and
initiatives. Getting a holistic understanding of riders' experience through the
feedback shared in those instruments is often challenging, mostly due to the
open-ended, unstructured nature of text feedback. In this paper, we propose
leveraging traditional transit CRM feedback to develop and deploy a
transit-topic-aware large language model (LLM) capable of classifying
open-ended text feedback to relevant transit-specific topics. First, we utilize
semi-supervised learning to engineer a training dataset of 11 broad transit
topics detected in a corpus of 6 years of customer feedback provided to the
Washington Metropolitan Area Transit Authority (WMATA). We then use this
dataset to train and thoroughly evaluate a language model based on the RoBERTa
architecture. We compare our LLM, MetRoBERTa, to classical machine learning
approaches utilizing keyword-based and lexicon representations. Our model
outperforms those methods across all evaluation metrics, providing an average
topic classification accuracy of 90%. Finally, we provide a value proposition
of this work demonstrating how the language model, alongside additional text
processing tools, can be applied to add structure to open-ended text sources of
feedback like Twitter. The framework and results we present provide a pathway
for an automated, generalizable approach for ingesting, visualizing, and
reporting transit riders' feedback at scale, enabling agencies to better
understand and improve customer experience.
Related papers
- Transit Pulse: Utilizing Social Media as a Source for Customer Feedback and Information Extraction with Large Language Model [12.6020349733674]
We propose a novel approach to extracting and analyzing transit-related information.
Our method employs Large Language Models (LLM), specifically Llama 3, for a streamlined analysis.
Our results demonstrate the potential of LLMs to transform social media data analysis in the public transit domain.
arXiv Detail & Related papers (2024-10-19T07:08:40Z) - Leveraging Large Language Models for Enhancing Public Transit Services [12.6020349733674]
This paper explores the utilization of Large Language Models (LLMs) in the public transit system.
Three transit LLM applications are presented: Tweet Writer, Trip Advisor, and Policy Navigator.
arXiv Detail & Related papers (2024-10-18T03:33:47Z) - Text2Traj2Text: Learning-by-Synthesis Framework for Contextual Captioning of Human Movement Trajectories [11.303926633163117]
This paper presents Text2Traj2Text, a novel learning-by-caption framework for captioning possible contexts behind shopper's trajectory data in retail stores.
Our work will impact various retail applications that need better customer understanding, such as targeted advertising and inventory management.
arXiv Detail & Related papers (2024-09-19T11:30:09Z) - Towards Zero-Shot Multimodal Machine Translation [64.9141931372384]
We propose a method to bypass the need for fully supervised data to train multimodal machine translation systems.
Our method, called ZeroMMT, consists in adapting a strong text-only machine translation (MT) model by training it on a mixture of two objectives.
To prove that our method generalizes to languages with no fully supervised training data available, we extend the CoMMuTE evaluation dataset to three new languages: Arabic, Russian and Chinese.
arXiv Detail & Related papers (2024-07-18T15:20:31Z) - Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation [65.16137964758612]
We explore the use of long-context capabilities in large language models to create synthetic reading comprehension data from entire books.
Our objective is to test the capabilities of LLMs to analyze, understand, and reason over problems that require a detailed comprehension of long spans of text.
arXiv Detail & Related papers (2024-05-31T20:15:10Z) - AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large Language Models [34.82568259708465]
Allhands is an innovative analytic framework designed for large-scale feedback analysis through a natural language interface.
LLMs are large language models that enhance accuracy, robustness, generalization, and user-friendliness.
Allhands delivers comprehensive multi-modal responses, including text, code, tables, and images.
arXiv Detail & Related papers (2024-03-22T12:13:16Z) - UltraFeedback: Boosting Language Models with Scaled AI Feedback [99.4633351133207]
We present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset.
Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models.
arXiv Detail & Related papers (2023-10-02T17:40:01Z) - Improving Conversational Recommendation Systems via Counterfactual Data
Simulation [73.4526400381668]
Conversational recommender systems (CRSs) aim to provide recommendation services via natural language conversations.
Existing CRS approaches often suffer from the issue of insufficient training due to the scarcity of training data.
We propose a CounterFactual data simulation approach for CRS, named CFCRS, to alleviate the issue of data scarcity in CRSs.
arXiv Detail & Related papers (2023-06-05T12:48:56Z) - Rethinking the Evaluation for Conversational Recommendation in the Era
of Large Language Models [115.7508325840751]
The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs)
In this paper, we embark on an investigation into the utilization of ChatGPT for conversational recommendation, revealing the inadequacy of the existing evaluation protocol.
We propose an interactive Evaluation approach based on LLMs named iEvaLM that harnesses LLM-based user simulators.
arXiv Detail & Related papers (2023-05-22T15:12:43Z) - Intent Recognition in Conversational Recommender Systems [0.0]
We introduce a pipeline to contextualize the input utterances in conversations.
We then take the next step towards leveraging reverse feature engineering to link the contextualized input and learning model to support intent recognition.
arXiv Detail & Related papers (2022-12-06T11:02:42Z) - Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
We present the first application of the BERTSum model to conversational language.
We generate abstractive summaries of narrated instructional videos across a wide variety of topics.
We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
arXiv Detail & Related papers (2020-08-21T20:59:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.