Automating Customer Needs Analysis: A Comparative Study of Large Language Models in the Travel Industry

• URL: http://arxiv.org/abs/2404.17975v1
• Date: Sat, 27 Apr 2024 18:28:10 GMT
• Title: Automating Customer Needs Analysis: A Comparative Study of Large Language Models in the Travel Industry
• Authors: Simone Barandoni, Filippo Chiarello, Lorenzo Cascone, Emiliano Marrale, Salvatore Puccio,
• Abstract summary: Large Language Models (LLMs) have emerged as powerful tools for extracting valuable insights from vast amounts of textual data. In this study, we conduct a comparative analysis of LLMs for the extraction of travel customer needs from TripAdvisor posts. Our findings highlight the efficacy of opensource LLMs, particularly Mistral 7B, in achieving comparable performance to larger closed models.
• Score: 2.4244694855867275
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the rapidly evolving landscape of Natural Language Processing (NLP), Large Language Models (LLMs) have emerged as powerful tools for many tasks, such as extracting valuable insights from vast amounts of textual data. In this study, we conduct a comparative analysis of LLMs for the extraction of travel customer needs from TripAdvisor posts. Leveraging a diverse range of models, including both open-source and proprietary ones such as GPT-4 and Gemini, we aim to elucidate their strengths and weaknesses in this specialized domain. Through an evaluation process involving metrics such as BERTScore, ROUGE, and BLEU, we assess the performance of each model in accurately identifying and summarizing customer needs. Our findings highlight the efficacy of opensource LLMs, particularly Mistral 7B, in achieving comparable performance to larger closed models while offering affordability and customization benefits. Additionally, we underscore the importance of considering factors such as model size, resource requirements, and performance metrics when selecting the most suitable LLM for customer needs analysis tasks. Overall, this study contributes valuable insights for businesses seeking to leverage advanced NLP techniques to enhance customer experience and drive operational efficiency in the travel industry.

Related papers

• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• LLM-Select: Feature Selection with Large Language Models [64.5099482021597]
  Large language models (LLMs) are capable of selecting the most predictive features, with performance rivaling the standard tools of data science. Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place.
  arXiv  Detail & Related papers  (2024-07-02T22:23:40Z)
• Benchmarking Open-Source Language Models for Efficient Question Answering in Industrial Applications [0.0]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in tasks such as question answering (QA) This paper presents a comprehensive benchmarking study comparing open-source LLMs with their non-open-source counterparts on the task of question answering.
  arXiv  Detail & Related papers  (2024-06-19T17:11:51Z)
• AvaTaR: Optimizing LLM Agents for Tool-Assisted Knowledge Retrieval [93.96463520716759]
  Large language model (LLM) agents have demonstrated impressive capability in utilizing external tools and knowledge to boost accuracy and reduce hallucinations. Here, we introduce AvaTaR, a novel framework that optimize an LLM agent to effectively use the provided tools and improve its performance on a given task/domain. We find AvaTaR consistently outperforms state-of-the-art approaches across all four challenging tasks and exhibits strong generalization ability when applied to novel cases.
  arXiv  Detail & Related papers  (2024-06-17T04:20:02Z)
• Assessing the Performance of Chinese Open Source Large Language Models in Information Extraction Tasks [12.400599440431188]
  Information Extraction (IE) plays a crucial role in Natural Language Processing (NLP) Recent experiments focusing on English IE tasks have shed light on the challenges faced by Large Language Models (LLMs) in achieving optimal performance.
  arXiv  Detail & Related papers  (2024-06-04T08:00:40Z)
• Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutions [2.5179515260542544]
  Large Language Models (LLMs) have gained significant attention across academia and industry for their versatile applications in text generation, question answering, and text summarization. To quantify the performance, it's crucial to have a comprehensive grasp of existing metrics. This paper offers a comprehensive exploration of LLM evaluation from a metrics perspective, providing insights into the selection and interpretation of metrics currently in use.
  arXiv  Detail & Related papers  (2024-04-14T03:54:00Z)
• LLM In-Context Recall is Prompt Dependent [0.0]
  A model's ability to do this significantly influences its practical efficacy and dependability in real-world applications. This study demonstrates that an LLM's recall capability is not only contingent upon the prompt's content but also may be compromised by biases in its training data.
  arXiv  Detail & Related papers  (2024-04-13T01:13:59Z)
• Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models [41.38520841504846]
  Large language models (LLMs) can provide automated preference feedback solely from image inputs to guide decision-making. In this study, we train a multimodal LLM, termed CriticGPT, capable of understanding trajectory videos in robot manipulation tasks. Experimental evaluation of the algorithm's preference accuracy demonstrates its effective generalization ability to new tasks. Performance on Meta-World tasks reveals that CriticGPT's reward model efficiently guides policy learning, surpassing rewards based on state-of-the-art pre-trained representation models.
  arXiv  Detail & Related papers  (2024-02-22T03:14:03Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
  arXiv  Detail & Related papers  (2023-12-26T07:24:46Z)
• An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models [116.50367506746713]
  We present an empirical study of scaling LLaVA up to 33B and 65B/70B. We find that scaling LMM consistently enhances model performance and improves language capabilities. We hope that this study makes state-of-the-art LMM research at a larger scale more accessible.
  arXiv  Detail & Related papers  (2023-09-18T17:30:46Z)
• A Survey on Large Language Models for Recommendation [77.91673633328148]
  Large Language Models (LLMs) have emerged as powerful tools in the field of Natural Language Processing (NLP) This survey presents a taxonomy that categorizes these models into two major paradigms, respectively Discriminative LLM for Recommendation (DLLM4Rec) and Generative LLM for Recommendation (GLLM4Rec)
  arXiv  Detail & Related papers  (2023-05-31T13:51:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.