HICM: An approach towards Harmonizing Indian Census Migration data and its applications

• URL: http://arxiv.org/abs/2604.12324v1
• Date: Tue, 14 Apr 2026 05:58:27 GMT
• Title: HICM: An approach towards Harmonizing Indian Census Migration data and its applications
• Abstract summary: This paper presents a data-centric framework, HICM, for harmonizing Indian census migration data recorded under the Indian census.<n>We identify two types of bias across three decades of migration data: measurement bias and representativeness bias.<n>An empirical evaluation of harmonized Indian interstate migration data reveals that bias-aware data correction substantially improves the consistency in the structure of the data.
• Score: 2.12833538322985
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliable analysis of migration is critically dependent on the quality and consistency of the underlying data. Indian migration data, primarily derived from decennial census records, are affected by systematic gaps arising from uneven coverage and measurement inconsistencies across states and time. This paper presents a data-centric framework, HICM, for harmonizing Indian census migration data recorded under the Indian census and correcting prominent sources of bias prior to downstream analyses. We explicitly identify two types of bias across three decades of migration data: measurement bias and representativeness bias. We propose to address these gaps through principled pre-processing, mitigation, and validation strategies grounded in statistical diagnostics. An empirical evaluation of harmonized Indian interstate migration data reveals that bias-aware data correction substantially improves the consistency in the structure of the data and enhances the reliability of subsequent temporal analysis results. By improving data quality through reproducible data imputation and smoothing, this work advances migration analytics and provides a robust foundation for policy-relevant longitudinal network analysis of Indian internal migration.

Related papers

• Joint Utilization of Geospatial and census proxies for Autoencoder-Assisted Downscaling (JUGAAD) of socioeconomic indicators in India [6.076588686970015]
  This study introduces a deep learning framework, JuGAAD, using Indian census and survey data from 2001 and 2011 as a case study.<n>We employ a three-step process: census and geospatial data are averaged into intermediate village-cluster-scale tessellations to reduce noise and regularize administrative boundary changes.<n>An autoencoder compresses high-dimensional National Sample Survey Office (NSSO) data into a low-dimensional latent representation; and a regression model maps upscaled census and geospatial data to this representation.<n>This function is applied to fine-grained census data to generate high-resolution predictions, validated against ground-
  arXiv  Detail & Related papers  (2026-07-17T21:57:55Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Data Fusion for Partial Identification of Causal Effects [62.56890808004615]
  We propose a novel partial identification framework that enables researchers to answer key questions.<n>Is the causal effect positive or negative? and How severe must assumption violations be to overturn this conclusion?<n>We apply our framework to the Project STAR study, which investigates the effect of classroom size on students' third-grade standardized test performance.
  arXiv  Detail & Related papers  (2025-05-30T07:13:01Z)
• Inferring fine-grained migration patterns across the United States [1.6594124470436404]
  We develop a scalable iterative-proportional-fitting based method that reconciles high-resolution but biased proprietary data with low-resolution but more reliable Census data.<n>We produce MIGRATE, a dataset of annual migration matrices from 2010 - 2019 that captures flows between 47.4 billion pairs of Census Block Groups.<n>These estimates are highly correlated with external ground-truth datasets, and improve accuracy and reduce bias relative to raw proprietary data.
  arXiv  Detail & Related papers  (2025-03-26T21:07:44Z)
• A Highly Granular Temporary Migration Dataset Derived From Mobile Phone Data in Senegal [0.0]
  This article introduces a detailed and open-access dataset that leverages mobile phone data to capture temporary migration in Senegal. The article presents a suite of methodological tools that not only include algorithmic methods for the detection of temporary migration events in digital traces, but also addresses key challenges in aggregating individual trajectories into coherent migration statistics.
  arXiv  Detail & Related papers  (2024-06-21T14:58:28Z)
• Can We Enhance the Quality of Mobile Crowdsensing Data Without Ground Truth? [45.875832406278214]
  This article proposes a prediction- and reputation-based truth discovery framework.<n>It can separate low-quality data from high-quality data in sensing tasks.<n>It outperforms existing methods in terms of identification accuracy and data quality enhancement.
  arXiv  Detail & Related papers  (2024-05-29T03:16:12Z)
• DAGnosis: Localized Identification of Data Inconsistencies using Structures [73.39285449012255]
  Identification and appropriate handling of inconsistencies in data at deployment time is crucial to reliably use machine learning models. We use directed acyclic graphs (DAGs) to encode the training set's features probability distribution and independencies as a structure. Our method, called DAGnosis, leverages these structural interactions to bring valuable and insightful data-centric conclusions.
  arXiv  Detail & Related papers  (2024-02-26T11:29:16Z)
• Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics [58.50754318846996]
  In this paper, we show that the performances of metrics are sensitive to data. The ranking of metrics varies when the evaluation is conducted on different datasets.
  arXiv  Detail & Related papers  (2022-03-29T18:58:28Z)
• Data-SUITE: Data-centric identification of in-distribution incongruous examples [81.21462458089142]
  Data-SUITE is a data-centric framework to identify incongruous regions of in-distribution (ID) data. We empirically validate Data-SUITE's performance and coverage guarantees.
  arXiv  Detail & Related papers  (2022-02-17T18:58:31Z)
• Leveraging Mobile Phone Data for Migration Flows [5.0161988361764775]
  Statistics on migration flows are often derived from census data, which suffer from intrinsic limitations. Alternative data sources, such as surveys and field observations, also suffer from reliability, costs, and scale limitations. The ubiquity of mobile phones enables an accurate and efficient collection of up-to-date data related to migration.
  arXiv  Detail & Related papers  (2021-05-31T13:41:47Z)
• Forecasting asylum-related migration flows with machine learning and data at scale [0.0]
  We show that adaptive machine learning algorithms can effectively forecast asylum-related migration flows. We exploit three tiers of data - geolocated events and internet searches in countries of origin, detections of irregular crossings at the EU border, and asylum recognition rates in countries of destination.
  arXiv  Detail & Related papers  (2020-11-09T11:31:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.