Socio-Conformal Calibration in Complex Survey Data: Marginal Validity Is Not Enough for Subgroup Reliability

• URL: http://arxiv.org/abs/2605.05562v1
• Date: Thu, 07 May 2026 01:10:48 GMT
• Title: Socio-Conformal Calibration in Complex Survey Data: Marginal Validity Is Not Enough for Subgroup Reliability
• Abstract summary: We study ordinal conformal prediction for five-level AI-attitude forecasting on the Pew American Trends Panel.<n>Standard conformal achieves nominal marginal coverage for all four base predictors but leaves weighted subgroup gaps of 13 percentage points.<n>For the strongest predictor (XGBoost), Mondrian worsens the fairness-efficiency trade-off.<n>A regularized comparator that shrinks group thresholds toward the global quantile mitigates this instability.
• Score: 1.089614199781423
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine-learning systems used in survey-based social measurement require uncertainty estimates that are reliable across population subgroups, not merely valid in aggregate. We study ordinal conformal prediction for five-level AI-attitude forecasting on the Pew American Trends Panel (Wave 152; n=4,591; 12 race x education subgroups), comparing standard split conformal, Mondrian (group-specific) conformal, and a regularized Mondrian comparator across 100 respondent-disjoint splits with survey-weighted evaluation. Standard conformal achieves nominal marginal coverage for all four base predictors but leaves weighted subgroup gaps of ~13 percentage points. For the strongest predictor (XGBoost), Mondrian worsens the fairness-efficiency trade-off: weighted set size rises by +0.036 (dz =1.66) while the weighted subgroup gap grows by +0.013 (dz =0.30). A regularized comparator that shrinks group thresholds toward the global quantile mitigates this instability (Delta gap = -0.001, Delta size = +0.012) but does not yield a decisive fairness gain. Failure analysis traces the mechanism to calibration-cell fragmentation interacting with group-specific confidence mismatch. The negative result persists across alternate outcome codings and subgroup granularities, demonstrating that nominal marginal validity is insufficient for subgroup reliability and that naive group-specific calibration is not a dependable fairness remedy in complex survey settings.

Related papers

• The Resolution of Causal Heterogeneity [5.222979968110479]
  Causal subgroup analyses often report a small number of groups summarizing treatment effect heterogeneity, as if that number were a well-defined estimand.<n>We replace it with a new population estimand, the resolution profile, a functional of the causal feature law.<n>Inference is organized around one cross-fitted Bayesian-bootstrap posterior for a single structured moment process.
  arXiv  Detail & Related papers  (2026-07-19T14:56:06Z)
• Towards Provably Fair Machine Learning: Bayesian Approaches For Consistent and Transparent Predictions [0.0]
  We present a principled algorithm for consistency in subgroup predictions.<n>Our classifier achieves zero consistency error by construction while exceeding baseline accuracy.<n>Minority demographics are disproportionately concentrated in small subgroups, precisely where frequentist inference is least reliable.
  arXiv  Detail & Related papers  (2026-06-10T19:16:26Z)
• On the Burden of Achieving Fairness in Conformal Prediction [8.720026908704371]
  We show that pooled calibration incurs irreducible group-wise coverage distortion at a scale set by cross-group quantile heterogeneity.<n>We quantify the cost of moving between policies which treat groups separately or pool them.
  arXiv  Detail & Related papers  (2026-05-14T02:02:06Z)
• Composite Silhouette: A Subsampling-based Aggregation Strategy [5.70259485914933]
  We introduce Composite Silhouette, an internal criterion for cluster-count selection.<n>It aggregates evidence across repeated subsampled clusterings rather than relying on a single partition.<n>Experiments on synthetic and real-world datasets show that Composite Silhouette effectively reconciles the strengths of micro- and macro-averaging.
  arXiv  Detail & Related papers  (2026-04-15T12:49:09Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness [61.45587642780908]
  We propose a three-step approach for parameter-efficient fine-tuning of image-text foundation models.<n>Our method improves its two key components: minority samples identification and the robust training algorithm.<n>Our theoretical analysis shows that our PPA enhances minority group identification and is Bayes optimal for minimizing the balanced group error.
  arXiv  Detail & Related papers  (2025-03-12T15:46:12Z)
• Adaptive Uncertainty Quantification for Generative AI [0.0]
  Mirroring split-conformal inference, we design a wrapper around a black-box algorithm which calibrates conformity scores.<n> adaptive partitioning is achieved by fitting a robust regression tree to the conformity scores on the calibration set.<n>Unlike traditional split-conformal inference, adaptive splitting and within-group calibration yields adaptive bands which can stretch and shrink locally.
  arXiv  Detail & Related papers  (2024-08-16T19:37:33Z)
• How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalance [64.1656365676171]
  Group imbalance has been a known problem in empirical risk minimization. This paper quantifies the impact of individual groups on the sample complexity, the convergence rate, and the average and group-level testing performance.
  arXiv  Detail & Related papers  (2024-03-12T04:38:05Z)
• Beyond calibration: estimating the grouping loss of modern neural networks [68.8204255655161]
  Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings.
  arXiv  Detail & Related papers  (2022-10-28T07:04:20Z)
• Fair admission risk prediction with proportional multicalibration [0.16249424686052708]
  Multicalibration constrains calibration error among flexibly-defined subpopulations. It is possible for a decision-maker to learn to trust or distrust model predictions for specific groups. We propose proportional multicalibration, a criteria that constrains the percent calibration error among groups and within prediction bins.
  arXiv  Detail & Related papers  (2022-09-29T08:15:29Z)
• Selective Classification Can Magnify Disparities Across Groups [89.14499988774985]
  We find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities. Increasing abstentions can even decrease accuracies on some groups. We train distributionally-robust models that achieve similar full-coverage accuracies across groups and show that selective classification uniformly improves each group.
  arXiv  Detail & Related papers  (2020-10-27T08:51:30Z)
• Distribution-free binary classification: prediction sets, confidence intervals and calibration [106.50279469344937]
  We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting. We derive confidence intervals for binned probabilities for both fixed-width and uniform-mass binning. As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration.
  arXiv  Detail & Related papers  (2020-06-18T14:17:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.