Trustworthy Representation Learning via Information Funnels and Bottlenecks

• URL: http://arxiv.org/abs/2211.01446v2
• Date: Wed, 05 Nov 2025 15:35:30 GMT
• Title: Trustworthy Representation Learning via Information Funnels and Bottlenecks
• Authors: João Machado de Freitas, Bernhard C. Geiger,
• Abstract summary: We introduce Conditional Privacy Funnel with Side-information (CPFSI)<n>CPFSI is applicable in both fully and semi-supervised settings.<n>We show that CPFSI effectively balances competing objectives and frequently outperforms existing approaches.
• Score: 6.750492479021436
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ensuring trustworthiness in machine learning -- by balancing utility, fairness, and privacy -- remains a critical challenge, particularly in representation learning. In this work, we investigate a family of closely related information-theoretic objectives, including information funnels and bottlenecks, designed to extract invariant representations from data. We introduce the Conditional Privacy Funnel with Side-information (CPFSI), a novel formulation within this family, applicable in both fully and semi-supervised settings. Given the intractability of these objectives, we derive neural-network-based approximations via amortized variational inference. We systematically analyze the trade-offs between utility, invariance, and representation fidelity, offering new insights into the Pareto frontiers of these methods. Our results demonstrate that CPFSI effectively balances these competing objectives and frequently outperforms existing approaches. Furthermore, we show that by intervening on sensitive attributes in CPFSI's predictive posterior enhances fairness while maintaining predictive performance. Finally, we focus on the real-world applicability of these approaches, particularly for learning robust and fair representations from tabular datasets in data scarce-environments -- a modality where these methods are often especially relevant.

Related papers

• Reliable and Reproducible Demographic Inference for Fairness in Face Analysis [63.46525489354455]
  We propose a fully reproducible DAI pipeline that replaces conventional end-to-end training with a modular transfer learning approach.<n>We audit this pipeline across three dimensions: accuracy, fairness, and a newly introduced notion of robustness, defined via intra-identity consistency.<n>Our results show that the proposed method outperforms strong baselines, particularly on ethnicity, which is the more challenging attribute.
  arXiv  Detail & Related papers  (2025-10-23T12:22:02Z)
• FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learning [103.45987800174724]
  Federated Learning (FL) confronts a significant challenge known as data heterogeneity, which impairs model performance and convergence.<n>We propose textbfFedGPS, a novel framework that seamlessly integrates statistical distribution and gradient information from others.
  arXiv  Detail & Related papers  (2025-10-23T06:10:11Z)
• Fairness Regularization in Federated Learning [1.4773243280881763]
  Federated Learning (FL) has emerged as a vital paradigm in modern machine learning.<n>This work focuses on performance equitable fairness, which aims to minimize differences in performance across clients.<n>We empirically show that FairGrad (approximate) and FairGrad* (exact) improve both fairness and overall model performance in heterogeneous data settings.
  arXiv  Detail & Related papers  (2025-08-16T13:32:41Z)
• Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence [92.07601770031236]
  We investigate semantically meaningful patterns in the attention heads of an encoder-only Transformer architecture. We find that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization.
  arXiv  Detail & Related papers  (2024-09-20T07:41:47Z)
• ARBEx: Attentive Feature Extraction with Reliability Balancing for Robust Facial Expression Learning [5.648318448953635]
  ARBEx is a novel attentive feature extraction framework driven by Vision Transformer. We employ learnable anchor points in the embedding space with label distributions and multi-head self-attention mechanism to optimize performance against weak predictions. Our strategy outperforms current state-of-the-art methodologies, according to extensive experiments conducted in a variety of contexts.
  arXiv  Detail & Related papers  (2023-05-02T15:10:01Z)
• Leveraging sparse and shared feature activations for disentangled representation learning [112.22699167017471]
  We propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation. We validate our approach on six real world distribution shift benchmarks, and different data modalities.
  arXiv  Detail & Related papers  (2023-04-17T01:33:24Z)
• Ignorance is Bliss: Robust Control via Information Gating [60.17644038829572]
  Informational parsimony provides a useful inductive bias for learning representations that achieve better generalization by being robust to noise and spurious correlations. We propose textitinformation gating as a way to learn parsimonious representations that identify the minimal information required for a task.
  arXiv  Detail & Related papers  (2023-03-10T18:31:50Z)
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [61.94125052118442]
  Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL) In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
  arXiv  Detail & Related papers  (2023-03-03T16:12:59Z)
• Variational Distillation for Multi-View Learning [104.17551354374821]
  We design several variational information bottlenecks to exploit two key characteristics for multi-view representation learning. Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels.
  arXiv  Detail & Related papers  (2022-06-20T03:09:46Z)
• RevUp: Revise and Update Information Bottleneck for Event Representation [16.54912614895861]
  In machine learning, latent variables play a key role to capture the underlying structure of data, but they are often unsupervised. We propose a semi-supervised information bottleneck-based model that enables the use of side knowledge to direct the learning of discrete latent variables. We show that our approach generalizes an existing method of parameter injection, and perform an empirical case study of our approach on language-based event modeling.
  arXiv  Detail & Related papers  (2022-05-24T17:54:59Z)
• Information-Theoretic Odometry Learning [83.36195426897768]
  We propose a unified information theoretic framework for learning-motivated methods aimed at odometry estimation. The proposed framework provides an elegant tool for performance evaluation and understanding in information-theoretic language.
  arXiv  Detail & Related papers  (2022-03-11T02:37:35Z)
• Learning Bias-Invariant Representation by Cross-Sample Mutual Information Minimization [77.8735802150511]
  We propose a cross-sample adversarial debiasing (CSAD) method to remove the bias information misused by the target task. The correlation measurement plays a critical role in adversarial debiasing and is conducted by a cross-sample neural mutual information estimator. We conduct thorough experiments on publicly available datasets to validate the advantages of the proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-08-11T21:17:02Z)
• Fair Representation Learning using Interpolation Enabled Disentanglement [9.043741281011304]
  We propose a novel method to address two key issues: (a) Can we simultaneously learn fair disentangled representations while ensuring the utility of the learned representation for downstream tasks, and (b)Can we provide theoretical insights into when the proposed approach will be both fair and accurate. To address the former, we propose the method FRIED, Fair Representation learning using Interpolation Enabled Disentanglement.
  arXiv  Detail & Related papers  (2021-07-31T17:32:12Z)
• Robust Representation Learning via Perceptual Similarity Metrics [18.842322467828502]
  Contrastive Input Morphing (CIM) is a representation learning framework that learns input-space transformations of the data. We show that CIM is complementary to other mutual information-based representation learning techniques.
  arXiv  Detail & Related papers  (2021-06-11T21:45:44Z)
• Conditional Contrastive Learning: Removing Undesirable Information in Self-Supervised Representations [108.29288034509305]
  We develop conditional contrastive learning to remove undesirable information in self-supervised representations. We demonstrate empirically that our methods can successfully learn self-supervised representations for downstream tasks.
  arXiv  Detail & Related papers  (2021-06-05T10:51:26Z)
• Nonlinear Invariant Risk Minimization: A Causal Approach [5.63479133344366]
  We propose a learning paradigm that enables out-of-distribution generalization in the nonlinear setting. We show identifiability of the data representation up to very simple transformations. Extensive experiments on both synthetic and real-world datasets show that our approach significantly outperforms a variety of baseline methods.
  arXiv  Detail & Related papers  (2021-02-24T15:38:41Z)
• Accurate and Robust Feature Importance Estimation under Distribution Shifts [49.58991359544005]
  PRoFILE is a novel feature importance estimation method. We show significant improvements over state-of-the-art approaches, both in terms of fidelity and robustness.
  arXiv  Detail & Related papers  (2020-09-30T05:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.