Differential Subgroup Discovery: Characterizing Where Two Populations Differ, and Why

• URL: http://arxiv.org/abs/2604.27741v1
• Date: Thu, 30 Apr 2026 11:31:42 GMT
• Title: Differential Subgroup Discovery: Characterizing Where Two Populations Differ, and Why
• Abstract summary: We study the problem of understanding where two populations differ within a feature space.<n> Differential subgroups reveal the regions of the feature space where population-level gaps are most pronounced.
• Score: 36.94047915240195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the problem of understanding where two populations differ within a feature space, which we formalize in the concept of a differential subgroup: a subset of individuals from both populations who, despite sharing similar characteristics, exhibit exceptional differences in a target outcome. Differential subgroups reveal the regions of the feature space where population-level gaps are most pronounced and can help practitioners identify the covariate combinations that are structurally responsible for these differences, e.g.~in clinical analysis, model diagnostics, or treatment-effect studies. We introduce a general optimization objective for discovering differential subgroups and establish conditions under which the resulting subgroups admit a causal interpretation of population differences. We propose DiffSub, a gradient-based approach that discovers interpretable differential subgroups in tabular data. Across synthetic benchmarks, medical case studies, model-error analyses, and treatment-effect settings, DiffSub identifies informative subgroups that reveal where population differences arise and why.

Related papers

• Conditional Hypothesis Generation for LLM-Based Text Analysis with Researcher-Specified Covariates [11.185598864391103]
  We introduce conditional hypothesis generation to steer hypothesis discovery toward differences within relevant subgroups.<n>Two challenges arise: the target subgroup may be underrepresented, and the direction of a difference may reverse across subgroups.<n>We propose two econometrics-inspired methods to detect sign reversals, and the other applies within-stratum demeaning and inverse-frequency reweighting to equalize underrepresented strata.
  arXiv  Detail & Related papers  (2026-06-02T02:07:46Z)
• Identifying latent disease factors differently expressed in patient subgroups using group factor analysis [54.67330718129736]
  We propose a novel approach to uncover subgroup-specific and subgroup-common latent factors. The proposed approach, sparse Group Factor Analysis (GFA) with regularised horseshoe priors, was implemented with probabilistic programming.
  arXiv  Detail & Related papers  (2024-10-10T13:12:14Z)
• Group-specific discriminant analysis reveals statistically validated sex differences in lateralization of brain functional network [7.01314545449491]
  Group-Specific Discriminant Analysis (GSDA) developed for first-order classification. Group-Specific Discriminant Analysis (GSDA) developed for first-order classification.
  arXiv  Detail & Related papers  (2024-04-08T16:36:07Z)
• A structured regression approach for evaluating model performance across intersectional subgroups [53.91682617836498]
  Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups. We introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups.
  arXiv  Detail & Related papers  (2024-01-26T14:21:45Z)
• The Role of Subgroup Separability in Group-Fair Medical Image Classification [18.29079361470428]
  We find a relationship between subgroup separability, subgroup disparities, and performance degradation when models are trained on data with systematic bias such as underdiagnosis. Our findings shed new light on the question of how models become biased, providing important insights for the development of fair medical imaging AI.
  arXiv  Detail & Related papers  (2023-07-06T06:06:47Z)
• Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization? [90.35044668396591]
  A recurring theme in machine learning is algorithmic monoculture: the same systems, or systems that share components, are deployed by multiple decision-makers. We propose the component-sharing hypothesis: if decision-makers share components like training data or specific models, then they will produce more homogeneous outcomes. We test this hypothesis on algorithmic fairness benchmarks, demonstrating that sharing training data reliably exacerbates homogenization. We conclude with philosophical analyses of and societal challenges for outcome homogenization, with an eye towards implications for deployed machine learning systems.
  arXiv  Detail & Related papers  (2022-11-25T09:33:11Z)
• Adaptive Identification of Populations with Treatment Benefit in Clinical Trials: Machine Learning Challenges and Solutions [78.31410227443102]
  We study the problem of adaptively identifying patient subpopulations that benefit from a given treatment during a confirmatory clinical trial. We propose AdaGGI and AdaGCPI, two meta-algorithms for subpopulation construction.
  arXiv  Detail & Related papers  (2022-08-11T14:27:49Z)
• Post-discovery Analysis of Anomalous Subsets [6.118940071203314]
  We propose an analysis of differentiated output (rather than discovery) and quantify anomalousness similarly to the counter-factual setting. Our approach is presented and the evaluation results on the 2019 MarketScan Commercial Claims and Medicare data, show that extra insights can be obtained.
  arXiv  Detail & Related papers  (2021-11-23T20:30:02Z)
• Recommendations for Bayesian hierarchical model specifications for case-control studies in mental health [0.0]
  Researchers must choose whether to assume all subjects are drawn from a common population, or to model them as deriving from separate populations. We ran systematic simulations on synthetic multi-group behavioural data from a commonly used bandit task. We found that fitting groups separately provided the most accurate and robust inference across all conditions.
  arXiv  Detail & Related papers  (2020-11-03T14:19:59Z)
• LOGAN: Local Group Bias Detection by Clustering [86.38331353310114]
  We argue that evaluating bias at the corpus level is not enough for understanding how biases are embedded in a model. We propose LOGAN, a new bias detection technique based on clustering. Experiments on toxicity classification and object classification tasks show that LOGAN identifies bias in a local region.
  arXiv  Detail & Related papers  (2020-10-06T16:42:51Z)
• Robust Recursive Partitioning for Heterogeneous Treatment Effects with Uncertainty Quantification [84.53697297858146]
  Subgroup analysis of treatment effects plays an important role in applications from medicine to public policy to recommender systems. Most of the current methods of subgroup analysis begin with a particular algorithm for estimating individualized treatment effects (ITE) This paper develops a new method for subgroup analysis, R2P, that addresses all these weaknesses.
  arXiv  Detail & Related papers  (2020-06-14T14:50:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.