Theoretical Analysis of Learned Database Operations under Distribution Shift through Distribution Learnability

• URL: http://arxiv.org/abs/2411.06241v1
• Date: Sat, 09 Nov 2024 17:47:05 GMT
• Title: Theoretical Analysis of Learned Database Operations under Distribution Shift through Distribution Learnability
• Authors: Sepanta Zeighami, Cyrus Shahahbi,
• Abstract summary: We present the first known theoretical characterization of the performance of learned models in dynamic datasets. Our results show novel theoretical characteristics achievable by learned models and provide bounds on the performance of the models. Our analysis develops the distribution learnability framework and novel theoretical tools which build the foundation for the analysis of learned database operations in the future.
• Score: 0.8594140167290097
• License:
• Abstract: Use of machine learning to perform database operations, such as indexing, cardinality estimation, and sorting, is shown to provide substantial performance benefits. However, when datasets change and data distribution shifts, empirical results also show performance degradation for learned models, possibly to worse than non-learned alternatives. This, together with a lack of theoretical understanding of learned methods undermines their practical applicability, since there are no guarantees on how well the models will perform after deployment. In this paper, we present the first known theoretical characterization of the performance of learned models in dynamic datasets, for the aforementioned operations. Our results show novel theoretical characteristics achievable by learned models and provide bounds on the performance of the models that characterize their advantages over non-learned methods, showing why and when learned models can outperform the alternatives. Our analysis develops the distribution learnability framework and novel theoretical tools which build the foundation for the analysis of learned database operations in the future.

Related papers

• Fitting Multiple Machine Learning Models with Performance Based Clustering [8.763425474439552]
  Traditional machine learning approaches assume that data comes from a single generating mechanism, which may not hold for most real life data. We introduce a clustering framework that eliminates this assumption by grouping the data according to the relations between the features and the target values. We extend our framework to applications having streaming data where we produce outcomes using an ensemble of models.
  arXiv  Detail & Related papers  (2024-11-10T19:38:35Z)
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• DETAIL: Task DEmonsTration Attribution for Interpretable In-context Learning [75.68193159293425]
  In-context learning (ICL) allows transformer-based language models to learn a specific task with a few "task demonstrations" without updating their parameters. We propose an influence function-based attribution technique, DETAIL, that addresses the specific characteristics of ICL. We experimentally prove the wide applicability of DETAIL by showing our attribution scores obtained on white-box models are transferable to black-box models in improving model performance.
  arXiv  Detail & Related papers  (2024-05-22T15:52:52Z)
• Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases [17.431381376675432]
  In this paper we explore the performance of large language models based on instruction tuning across different scales of instruction data. With Bloomz-7B1-mt as the base model, the results show that merely increasing the amount of instruction data leads to continuous improvement in tasks such as open-ended generation. We propose potential future research directions such as effectively selecting high-quality training data, scaling base models and training methods specialized for hard tasks.
  arXiv  Detail & Related papers  (2023-03-26T14:49:37Z)
• Minimal Value-Equivalent Partial Models for Scalable and Robust Planning in Lifelong Reinforcement Learning [56.50123642237106]
  Common practice in model-based reinforcement learning is to learn models that model every aspect of the agent's environment. We argue that such models are not particularly well-suited for performing scalable and robust planning in lifelong reinforcement learning scenarios. We propose new kinds of models that only model the relevant aspects of the environment, which we call "minimal value-minimal partial models"
  arXiv  Detail & Related papers  (2023-01-24T16:40:01Z)
• Striving for data-model efficiency: Identifying data externalities on group performance [75.17591306911015]
  Building trustworthy, effective, and responsible machine learning systems hinges on understanding how differences in training data and modeling decisions interact to impact predictive performance. We focus on a particular type of data-model inefficiency, in which adding training data from some sources can actually lower performance evaluated on key sub-groups of the population. Our results indicate that data-efficiency is a key component of both accurate and trustworthy machine learning.
  arXiv  Detail & Related papers  (2022-11-11T16:48:27Z)
• MACE: An Efficient Model-Agnostic Framework for Counterfactual Explanation [132.77005365032468]
  We propose a novel framework of Model-Agnostic Counterfactual Explanation (MACE) In our MACE approach, we propose a novel RL-based method for finding good counterfactual examples and a gradient-less descent method for improving proximity. Experiments on public datasets validate the effectiveness with better validity, sparsity and proximity.
  arXiv  Detail & Related papers  (2022-05-31T04:57:06Z)
• Learnability of Learning Performance and Its Application to Data Valuation [11.78594243870616]
  In most machine learning (ML) tasks, evaluating learning performance on a given dataset requires intensive computation. The ability to efficiently estimate learning performance may benefit a wide spectrum of applications, such as active learning, data quality management, and data valuation. Recent empirical studies show that for many common ML models, one can accurately learn a parametric model that predicts learning performance for any given input datasets using a small amount of samples.
  arXiv  Detail & Related papers  (2021-07-13T18:56:04Z)
• Model Embedding Model-Based Reinforcement Learning [4.566180616886624]
  Model-based reinforcement learning (MBRL) has shown its advantages in sample-efficiency over model-free reinforcement learning (MFRL) Despite the impressive results it achieves, it still faces a trade-off between the ease of data generation and model bias. We propose a simple and elegant model-embedding model-based reinforcement learning (MEMB) algorithm in the framework of the probabilistic reinforcement learning.
  arXiv  Detail & Related papers  (2020-06-16T15:10:28Z)
• How Training Data Impacts Performance in Learning-based Control [67.7875109298865]
  This paper derives an analytical relationship between the density of the training data and the control performance. We formulate a quality measure for the data set, which we refer to as $rho$-gap. We show how the $rho$-gap can be applied to a feedback linearizing control law.
  arXiv  Detail & Related papers  (2020-05-25T12:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.