Making LLMs Work for Enterprise Data Tasks

• URL: http://arxiv.org/abs/2407.20256v1
• Date: Mon, 22 Jul 2024 21:16:59 GMT
• Title: Making LLMs Work for Enterprise Data Tasks
• Authors: Çağatay Demiralp, Fabian Wenz, Peter Baile Chen, Moe Kayali, Nesime Tatbul, Michael Stonebraker,
• Abstract summary: Large language models (LLMs) know little about enterprise database tables in the private data ecosystem. As LLMs' performance is tied to their training data, a crucial question is how useful they can be in improving enterprise database management and analysis tasks.
• Score: 4.233865241818131
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) know little about enterprise database tables in the private data ecosystem, which substantially differ from web text in structure and content. As LLMs' performance is tied to their training data, a crucial question is how useful they can be in improving enterprise database management and analysis tasks. To address this, we contribute experimental results on LLMs' performance for text-to-SQL and semantic column-type detection tasks on enterprise datasets. The performance of LLMs on enterprise data is significantly lower than on benchmark datasets commonly used. Informed by our findings and feedback from industry practitioners, we identify three fundamental challenges -- latency, cost, and quality -- and propose potential solutions to use LLMs in enterprise data workflows effectively.

Related papers

• Mind the Data Gap: Bridging LLMs to Enterprise Data Integration [2.7248990920379725]
  We show that the performance of methods based on large language models (LLMs) seriously degrades when tested on real-world datasets. We release a new benchmark dataset, the GOBY Benchmark, to advance discovery in enterprise data integration.
  arXiv  Detail & Related papers  (2024-12-29T03:07:20Z)
• Trustworthy and Efficient LLMs Meet Databases [9.243304683497904]
  Making large language models (LLMs) more trustworthy and efficient has gained significant attention. This tutorial explores such efforts and makes them transparent to the database community.
  arXiv  Detail & Related papers  (2024-12-23T22:34:40Z)
• Tackling prediction tasks in relational databases with LLMs [1.8434042562191815]
  This work addresses the notion that large language models (LLMs) cannot yield satisfactory results on relational databases. Using the recently introduced RelBench benchmark, we demonstrate that even a straightforward application of LLMs achieves competitive performance on these tasks.
  arXiv  Detail & Related papers  (2024-11-18T18:48:13Z)
• Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models [79.65071553905021]
  We propose Data Advisor, a method for generating data that takes into account the characteristics of the desired dataset. Data Advisor monitors the status of the generated data, identifies weaknesses in the current dataset, and advises the next iteration of data generation.
  arXiv  Detail & Related papers  (2024-10-07T17:59:58Z)
• 60 Data Points are Sufficient to Fine-Tune LLMs for Question-Answering [50.12622877002846]
  Large language models (LLMs) encode extensive world knowledge through pre-training on massive datasets, which can be fine-tuned for the question-answering (QA) task. We categorize supervised fine-tuning (SFT) data based on the extent of knowledge memorized by the pretrained LLMs. Our experiments show that as few as 60 data points during the SFT stage can activate the knowledge encoded during pre-training, enabling LLMs to perform the QA task.
  arXiv  Detail & Related papers  (2024-09-24T07:38:38Z)
• Relational Database Augmented Large Language Model [59.38841050766026]
  Large language models (LLMs) excel in many natural language processing (NLP) tasks. They can only incorporate new knowledge through training or supervised fine-tuning processes. This precise, up-to-date, and private information is typically stored in relational databases.
  arXiv  Detail & Related papers  (2024-07-21T06:19:10Z)
• Efficient Prompting for LLM-based Generative Internet of Things [88.84327500311464]
  Large language models (LLMs) have demonstrated remarkable capacities on various tasks, and integrating the capacities of LLMs into the Internet of Things (IoT) applications has drawn much research attention recently. Due to security concerns, many institutions avoid accessing state-of-the-art commercial LLM services, requiring the deployment and utilization of open-source LLMs in a local network setting. We propose a LLM-based Generative IoT (GIoT) system deployed in the local network setting in this study.
  arXiv  Detail & Related papers  (2024-06-14T19:24:00Z)
• LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement [79.31084387589968]
  Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. We propose LLM2LLM, a data augmentation strategy that uses a teacher LLM to enhance a small seed dataset. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime.
  arXiv  Detail & Related papers  (2024-03-22T08:57:07Z)
• Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation [33.41556606816004]
  Large Language Models (LLMs) have emerged as a powerful tool in advancing the Text-to- task. There is still no consensus on the optimal prompt templates and design frameworks. Existing benchmarks inadequately explore the performance of LLMs across the various sub-tasks of the Text-to- process.
  arXiv  Detail & Related papers  (2024-03-05T13:23:48Z)
• Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation [76.76046657162306]
  Large language models (LLMs) have emerged as a new paradigm for Text-to- task. Large language models (LLMs) have emerged as a new paradigm for Text-to- task.
  arXiv  Detail & Related papers  (2023-08-29T14:59:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.