FAIRWELL: Fair Multimodal Self-Supervised Learning for Wellbeing Prediction

• URL: http://arxiv.org/abs/2508.16748v1
• Date: Fri, 22 Aug 2025 19:03:06 GMT
• Title: FAIRWELL: Fair Multimodal Self-Supervised Learning for Wellbeing Prediction
• Authors: Jiaee Cheong, Abtin Mogharabin, Paul Liang, Hatice Gunes, Sinan Kalkan,
• Abstract summary: We propose a novel subject-level loss function to learn fairer representations via the following three mechanisms.<n>We evaluate our method on three challenging real-world heterogeneous healthcare datasets.
• Score: 16.913286294651236
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Early efforts on leveraging self-supervised learning (SSL) to improve machine learning (ML) fairness has proven promising. However, such an approach has yet to be explored within a multimodal context. Prior work has shown that, within a multimodal setting, different modalities contain modality-unique information that can complement information of other modalities. Leveraging on this, we propose a novel subject-level loss function to learn fairer representations via the following three mechanisms, adapting the variance-invariance-covariance regularization (VICReg) method: (i) the variance term, which reduces reliance on the protected attribute as a trivial solution; (ii) the invariance term, which ensures consistent predictions for similar individuals; and (iii) the covariance term, which minimizes correlational dependence on the protected attribute. Consequently, our loss function, coined as FAIRWELL, aims to obtain subject-independent representations, enforcing fairness in multimodal prediction tasks. We evaluate our method on three challenging real-world heterogeneous healthcare datasets (i.e. D-Vlog, MIMIC and MODMA) which contain different modalities of varying length and different prediction tasks. Our findings indicate that our framework improves overall fairness performance with minimal reduction in classification performance and significantly improves on the performance-fairness Pareto frontier.

Related papers

• Beyond I-Con: Exploring New Dimension of Distance Measures in Representation Learning [7.8851393122408515]
  We present Beyond I-Con, a framework that enables systematic discovery of novel loss functions.<n>Our results highlight the importance of considering divergence and similarity kernel choices in representation learning optimization.
  arXiv  Detail & Related papers  (2025-09-05T01:23:59Z)
• Unified modality separation: A vision-language framework for unsupervised domain adaptation [60.8391821117794]
  Unsupervised domain adaptation (UDA) enables models trained on a labeled source domain to handle new unlabeled domains.<n>We propose a unified modality separation framework that accommodates both modality-specific and modality-invariant components.<n>Our methods achieve up to 9% performance gain with 9 times of computational efficiencies.
  arXiv  Detail & Related papers  (2025-08-07T02:51:10Z)
• MIRRAMS: Learning Robust Tabular Models under Unseen Missingness Shifts [2.5357049657770516]
  Missing values often reflect variations in data collection policies, which may shift across time or locations.<n>Such shifts in the missingness distribution between training and test inputs pose a significant challenge to achieving robust predictive performance.<n>We propose a novel deep learning framework designed to address this challenge, particularly in the common yet challenging scenario where the test-time dataset is unseen.
  arXiv  Detail & Related papers  (2025-07-11T03:03:30Z)
• Confidence-Aware Self-Distillation for Multimodal Sentiment Analysis with Incomplete Modalities [15.205192581534973]
  Multimodal sentiment analysis aims to understand human sentiment through multimodal data.<n>Existing methods for handling modality missingness are based on data reconstruction or common subspace projections.<n>We propose a Confidence-Aware Self-Distillation (CASD) strategy that effectively incorporates multimodal probabilistic embeddings.
  arXiv  Detail & Related papers  (2025-06-02T09:48:41Z)
• Asymmetric Reinforcing against Multi-modal Representation Bias [59.685072206359855]
  We propose an Asymmetric Reinforcing method against Multimodal representation bias (ARM)<n>Our ARM dynamically reinforces the weak modalities while maintaining the ability to represent dominant modalities through conditional mutual information.<n>We have significantly improved the performance of multimodal learning, making notable progress in mitigating imbalanced multimodal learning.
  arXiv  Detail & Related papers  (2025-01-02T13:00:06Z)
• Chain-of-Thought Prompting for Demographic Inference with Large Multimodal Models [58.58594658683919]
  Large multimodal models (LMMs) have shown transformative potential across various research tasks. Our findings indicate LMMs possess advantages in zero-shot learning, interpretability, and handling uncurated 'in-the-wild' inputs. We propose a Chain-of-Thought augmented prompting approach, which effectively mitigates the off-target prediction issue.
  arXiv  Detail & Related papers  (2024-05-24T16:26:56Z)
• Maturity-Aware Active Learning for Semantic Segmentation with Hierarchically-Adaptive Sample Assessment [18.65352271757926]
  "Maturity-Aware Distribution Breakdown-based Active Learning" (MADBAL) is an AL method that takes into account the different "sample" definitions jointly. MADBAL makes significant performance leaps even in the early AL stage, hence reducing the training burden significantly. It outperforms state-of-the-art methods on Cityscapes and PASCAL VOC datasets as verified in our experiments.
  arXiv  Detail & Related papers  (2023-08-28T21:13:04Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)
• Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions [91.63716984911278]
  We introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks.
  arXiv  Detail & Related papers  (2021-11-11T14:28:12Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Metric-Learning-Assisted Domain Adaptation [18.62119154143642]
  Many existing domain alignment methods assume that a low source risk, together with the alignment of distributions of source and target, means a low target risk. We propose a novel metric-learning-assisted domain adaptation (MLA-DA) method, which employs a novel triplet loss for helping better feature alignment.
  arXiv  Detail & Related papers  (2020-04-23T04:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.