TabPATE: Differentially Private Tabular In-Context Learning Without Public Data

• URL: http://arxiv.org/abs/2606.31474v1
• Date: Tue, 30 Jun 2026 10:50:04 GMT
• Title: TabPATE: Differentially Private Tabular In-Context Learning Without Public Data
• Abstract summary: Tabular foundation models enable in-context learning from small labeled datasets, but the private records placed in context can leak through model predictions.<n>We introduce TabPATE, a differentially private PATE-style defense for tabular ICL that does not require public in-distribution data.
• Score: 41.262278295182966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tabular foundation models enable accurate in-context learning (ICL) from small labeled datasets, but the private records placed in context can leak through model predictions. We first show that even basic membership inference attacks succeed against tabular ICL, motivating formal privacy protection. We then introduce TabPATE, a differentially private PATE-style defense for tabular ICL that does not require public in-distribution data. TabPATE partitions the private context across teacher models, privately aggregates their labels on synthetic tabular queries, and releases the resulting labeled queries as a student context. Because tabular features are bounded and relatively low-dimensional, useful queries can be generated from feature ranges alone or from lightly privatized marginals. Across tabular benchmarks, TabPATE preserves competitive utility while reducing membership inference to near-random success, providing a practical path to private tabular ICL without public data.

Related papers

• Private PoEtry: Private In-Context Learning via Product of Experts [58.496468062236225]
  In-context learning (ICL) enables Large Language Models to adapt to new tasks with only a small set of examples at inference time.<n>Existing differential privacy approaches to ICL are either computationally expensive or rely on oversampling, synthetic data generation, or unnecessary thresholding.<n>We reformulate private ICL through the lens of a Product-of-Experts model. This gives a theoretically grounded framework, and the algorithm can be trivially parallelized.<n>We find that our method improves accuracy by more than 30 percentage points on average compared to prior DP-ICL methods, while maintaining strong privacy guarantees.
  arXiv  Detail & Related papers  (2026-02-04T19:56:24Z)
• ConTextTab: A Semantics-Aware Tabular In-Context Learner [1.0999592665107414]
  We introduce ConTextTab, integrating semantic understanding and alignment into a table-native ICL framework.<n>Our model is competitive with SOTA across a broad set of benchmarks while setting a new standard on the semantically rich CARTE benchmark.
  arXiv  Detail & Related papers  (2025-06-12T13:57:29Z)
• Do You Really Need Public Data? Surrogate Public Data for Differential Privacy on Tabular Data [10.1687640711587]
  This work introduces the notion of "surrogate" public data, which consume no privacy loss budget and are constructed solely from publicly available schema or metadata.<n>We automate the process of generating surrogate public data with large language models (LLMs)<n>In particular, we propose two methods: direct record generation as CSV files, and automated structural causal model (SCM) construction for sampling records.
  arXiv  Detail & Related papers  (2025-04-19T17:55:10Z)
• A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers [70.20477771578824]
  Existing approaches to event prediction include time-aware positional embeddings, learned row and field encodings, and oversampling methods for addressing class imbalance. We propose a simple but flexible baseline using standard autoregressive LLM-style transformers with elementary positional embeddings and a causal language modeling objective. Our baseline outperforms existing approaches across popular datasets and can be employed for various use-cases.
  arXiv  Detail & Related papers  (2024-10-14T15:59:16Z)
• Towards Split Learning-based Privacy-Preserving Record Linkage [49.1574468325115]
  Split Learning has been introduced to facilitate applications where user data privacy is a requirement. In this paper, we investigate the potentials of Split Learning for Privacy-Preserving Record Matching.
  arXiv  Detail & Related papers  (2024-09-02T09:17:05Z)
• DP-TabICL: In-Context Learning with Differentially Private Tabular Data [12.814878223075437]
  In-context learning (ICL) enables large language models (LLMs) to adapt to new tasks. LLMs can leak information contained in prompts. This work serves as an initial investigation into how to use differential privacy (DP) We formulate two private ICL frameworks with provable privacy guarantees in both the local (LDP-TabICL) and global (GDP-TabICL)
  arXiv  Detail & Related papers  (2024-03-08T21:19:01Z)
• Rethinking Pre-Training in Tabular Data: A Neighborhood Embedding Perspective [71.45945607871715]
  We propose Tabular data Pre-Training via Meta-representation (TabPTM)<n>The core idea is to embed data instances into a shared feature space, where each instance is represented by its distance to a fixed number of nearest neighbors and their labels.<n>Extensive experiments on 101 datasets confirm TabPTM's effectiveness in both classification and regression tasks, with and without fine-tuning.
  arXiv  Detail & Related papers  (2023-10-31T18:03:54Z)
• TabuLa: Harnessing Language Models for Tabular Data Synthesis [4.539846270369207]
  Tabula is a tabular data synthesizer that leverages the structure of large language models (LLMs)<n>Unlike state-of-the-art (SOTA) LLMs, Tabula discards the pre-trained weights originally designed for natural language tasks.<n> experiments show that Tabula achieves superior synthetic data utility compared to current SOTA methods.
  arXiv  Detail & Related papers  (2023-10-19T13:50:56Z)
• Privately generating tabular data using language models [80.67328256105891]
  Privately generating synthetic data from a table is an important brick of a privacy-first world. We propose and investigate a simple approach of treating each row in a table as a sentence and training a language model with differential privacy.
  arXiv  Detail & Related papers  (2023-06-07T21:53:14Z)
• STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables [64.0903766169603]
  We propose a framework for few-shot semi-supervised learning, coined Self-generated Tasks from UNlabeled Tables (STUNT) Our key idea is to self-generate diverse few-shot tasks by treating randomly chosen columns as a target label. We then employ a meta-learning scheme to learn generalizable knowledge with the constructed tasks.
  arXiv  Detail & Related papers  (2023-03-02T02:37:54Z)
• Differentially Private Label Protection in Split Learning [20.691549091238965]
  Split learning is a distributed training framework that allows multiple parties to jointly train a machine learning model over partitioned data. Recent works showed that the implementation of split learning suffers from severe privacy risks that a semi-honest adversary can easily reconstruct labels. We propose textsfTPSL (Transcript Private Split Learning), a generic gradient based split learning framework that provides provable differential privacy guarantee.
  arXiv  Detail & Related papers  (2022-03-04T00:35:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.