Multi-Task Differential Privacy Under Distribution Skew

• URL: http://arxiv.org/abs/2302.07975v1
• Date: Wed, 15 Feb 2023 22:44:43 GMT
• Title: Multi-Task Differential Privacy Under Distribution Skew
• Abstract summary: We study the problem of multi-task learning under user-level differential privacy. One important aspect of the problem, that can significantly impact quality, is the distribution skew among tasks.
• Score: 36.770164806995204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the problem of multi-task learning under user-level differential privacy, in which $n$ users contribute data to $m$ tasks, each involving a subset of users. One important aspect of the problem, that can significantly impact quality, is the distribution skew among tasks. Certain tasks may have much fewer data samples than others, making them more susceptible to the noise added for privacy. It is natural to ask whether algorithms can adapt to this skew to improve the overall utility. We give a systematic analysis of the problem, by studying how to optimally allocate a user's privacy budget among tasks. We propose a generic algorithm, based on an adaptive reweighting of the empirical loss, and show that when there is task distribution skew, this gives a quantifiable improvement of excess empirical risk. Experimental studies on recommendation problems that exhibit a long tail of small tasks, demonstrate that our methods significantly improve utility, achieving the state of the art on two standard benchmarks.

Related papers

• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Towards Model-Based Data Acquisition for Subjective Multi-Task NLP Problems [12.38430125789305]
  We propose a new model-based approach that allows the selection of tasks annotated individually for each text in a multi-task scenario. Experiments carried out on three datasets, dozens of NLP tasks, and thousands of annotations show that our method allows up to 40% reduction in the number of annotations with negligible loss of knowledge.
  arXiv  Detail & Related papers  (2023-12-13T15:03:27Z)
• Task Selection and Assignment for Multi-modal Multi-task Dialogue Act Classification with Non-stationary Multi-armed Bandits [11.682678945754837]
  Multi-task learning (MTL) aims to improve the performance of a primary task by jointly learning with related auxiliary tasks. Previous studies suggest that such a random selection of tasks may not be helpful, and can even be harmful to performance. This paper proposes a method for selecting and assigning tasks based on non-stationary multi-armed bandits.
  arXiv  Detail & Related papers  (2023-09-18T14:51:51Z)
• Label Budget Allocation in Multi-Task Learning [61.12008399759143]
  In multi-task learning, related tasks provide information to each other and improve overall performance. How should the label budget be allocated among different tasks to achieve optimal multi-task performance? We are the first to propose and formally define the label budget allocation problem in multi-task learning.
  arXiv  Detail & Related papers  (2023-08-24T17:38:14Z)
• Multi-task Bias-Variance Trade-off Through Functional Constraints [102.64082402388192]
  Multi-task learning aims to acquire a set of functions that perform well for diverse tasks. In this paper we draw intuition from the two extreme learning scenarios -- a single function for all tasks, and a task-specific function that ignores the other tasks. We introduce a constrained learning formulation that enforces domain specific solutions to a central function.
  arXiv  Detail & Related papers  (2022-10-27T16:06:47Z)
• Collaborative Anomaly Detection [66.51075412012581]
  We propose collaborative anomaly detection (CAD) to jointly learn all tasks with an embedding encoding correlations among tasks. We explore CAD with conditional density estimation and conditional likelihood ratio estimation. It is beneficial to select a small number of tasks in advance to learn a task embedding model, and then use it to warm-start all task embeddings.
  arXiv  Detail & Related papers  (2022-09-20T18:01:07Z)
• A new perspective on classification: optimally allocating limited resources to uncertain tasks [4.169130102668252]
  In credit card fraud detection, for instance, a bank can only assign a small subset of transactions to their fraud investigations team. We argue that using classification to address task uncertainty is inherently suboptimal as it does not take into account the available capacity. We present a novel solution using learning to rank by directly optimizing the assignment's expected profit given limited capacity.
  arXiv  Detail & Related papers  (2022-02-09T10:14:45Z)
• Learning with User-Level Privacy [61.62978104304273]
  We analyze algorithms to solve a range of learning tasks under user-level differential privacy constraints. Rather than guaranteeing only the privacy of individual samples, user-level DP protects a user's entire contribution. We derive an algorithm that privately answers a sequence of $K$ adaptively chosen queries with privacy cost proportional to $tau$, and apply it to solve the learning tasks we consider.
  arXiv  Detail & Related papers  (2021-02-23T18:25:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.