Probabilistic Hash Embeddings for Online Learning of Categorical Features

• URL: http://arxiv.org/abs/2511.20893v2
• Date: Sun, 30 Nov 2025 06:51:16 GMT
• Title: Probabilistic Hash Embeddings for Online Learning of Categorical Features
• Authors: Aodong Li, Abishek Sankararaman, Balakrishnan Narayanaswamy,
• Abstract summary: We study streaming data with categorical features where the vocabulary of categorical feature values is changing.<n>Hashing is commonly used to map these categorical values into a feature space of fixed size before learning their embeddings.<n>We show that deterministic embeddings are sensitive to the arrival order of categories and suffer from forgetting in online learning.
• Score: 15.264601005614145
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study streaming data with categorical features where the vocabulary of categorical feature values is changing and can even grow unboundedly over time. Feature hashing is commonly used as a pre-processing step to map these categorical values into a feature space of fixed size before learning their embeddings. While these methods have been developed and evaluated for offline or batch settings, in this paper we consider online settings. We show that deterministic embeddings are sensitive to the arrival order of categories and suffer from forgetting in online learning, leading to performance deterioration. To mitigate this issue, we propose a probabilistic hash embedding (PHE) model that treats hash embeddings as stochastic and applies Bayesian online learning to learn incrementally from data. Based on the structure of PHE, we derive a scalable inference algorithm to learn model parameters and infer/update the posteriors of hash embeddings and other latent variables. Our algorithm (i) can handle an evolving vocabulary of categorical items, (ii) is adaptive to new items without forgetting old items, (iii) is implementable with a bounded set of parameters that does not grow with the number of distinct observed values on the stream, and (iv) is invariant to the item arrival order. Experiments in classification, sequence modeling, and recommendation systems in online learning setups demonstrate the superior performance of PHE while maintaining high memory efficiency (consumes as low as 2~4 memory of a one-hot embedding table). Supplementary materials are at https://github.com/aodongli/probabilistic-hash-embeddings

Related papers

• Bayesian Test-Time Adaptation for Vision-Language Models [51.93247610195295]
  Test-time adaptation with pre-trained vision-language models, such as CLIP, aims to adapt the model to new, potentially out-of-distribution test data.<n>We propose a novel approach, textbfBayesian textbfClass textbfAdaptation (BCA), which in addition to continuously updating class embeddings to adapt likelihood, also uses the posterior of incoming samples to continuously update the prior for each class embedding.
  arXiv  Detail & Related papers  (2025-03-12T10:42:11Z)
• Adaptive Cross Batch Normalization for Metric Learning [75.91093210956116]
  Metric learning is a fundamental problem in computer vision. We show that it is equally important to ensure that the accumulated embeddings are up to date. In particular, it is necessary to circumvent the representational drift between the accumulated embeddings and the feature embeddings at the current training iteration.
  arXiv  Detail & Related papers  (2023-03-30T03:22:52Z)
• Learning Context-aware Classifier for Semantic Segmentation [88.88198210948426]
  In this paper, contextual hints are exploited via learning a context-aware classifier. Our method is model-agnostic and can be easily applied to generic segmentation models. With only negligible additional parameters and +2% inference time, decent performance gain has been achieved on both small and large models.
  arXiv  Detail & Related papers  (2023-03-21T07:00:35Z)
• ORFit: One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares [5.430441358049335]
  We investigate the problem of one-pass learning, in which a model is trained on sequentially arriving data without retraining on previous datapoints.<n>We propose Orthogonal Recursive Fitting (ORFit), an algorithm for one-pass learning which seeks to perfectly fit each new datapoint while minimally altering the predictions on previous datapoints.
  arXiv  Detail & Related papers  (2022-07-28T02:01:31Z)
• Continual Learning For On-Device Environmental Sound Classification [63.81276321857279]
  We propose a simple and efficient continual learning method for on-device environmental sound classification. Our method selects the historical data for the training by measuring the per-sample classification uncertainty.
  arXiv  Detail & Related papers  (2022-07-15T12:13:04Z)
• Learning to Hash Naturally Sorts [84.90210592082829]
  We introduce Naturally-Sorted Hashing (NSH) to train a deep hashing model with sorted results end-to-end. NSH sort the Hamming distances of samples' hash codes and accordingly gather their latent representations for self-supervised training. We describe a novel Sorted Noise-Contrastive Estimation (SortedNCE) loss that selectively picks positive and negative samples for contrastive learning.
  arXiv  Detail & Related papers  (2022-01-31T16:19:02Z)
• Deep Self-Adaptive Hashing for Image Retrieval [16.768754022585057]
  We propose a textbfDeep Self-Adaptive Hashing(DSAH) model to adaptively capture the semantic information with two special designs. First, we construct a neighborhood-based similarity matrix, and then refine this initial similarity matrix with a novel update strategy. Secondly, we measure the priorities of data pairs with PIC and assign adaptive weights to them, which is relies on the assumption that more dissimilar data pairs contain more discriminative information for hash learning.
  arXiv  Detail & Related papers  (2021-08-16T13:53:20Z)
• Scalable Optimal Classifiers for Adversarial Settings under Uncertainty [10.90668635921398]
  We consider the problem of finding optimal classifiers in an adversarial setting where the class-1 data is generated by an attacker whose objective is not known to the defender. We show that this low-dimensional characterization enables to develop a training method to compute provably approximately optimal classifiers in a scalable manner.
  arXiv  Detail & Related papers  (2021-06-28T13:33:53Z)
• Few-Shot Incremental Learning with Continually Evolved Classifiers [46.278573301326276]
  Few-shot class-incremental learning (FSCIL) aims to design machine learning algorithms that can continually learn new concepts from a few data points. The difficulty lies in that limited data from new classes not only lead to significant overfitting issues but also exacerbate the notorious catastrophic forgetting problems. We propose a Continually Evolved CIF ( CEC) that employs a graph model to propagate context information between classifiers for adaptation.
  arXiv  Detail & Related papers  (2021-04-07T10:54:51Z)
• Fast Class-wise Updating for Online Hashing [196.14748396106955]
  This paper presents a novel supervised online hashing scheme, termed Fast Class-wise Updating for Online Hashing (FCOH) A class-wise updating method is developed to decompose the binary code learning and alternatively renew the hash functions in a class-wise fashion, which well addresses the burden on large amounts of training batches. To further achieve online efficiency, we propose a semi-relaxation optimization, which accelerates the online training by treating different binary constraints independently.
  arXiv  Detail & Related papers  (2020-12-01T07:41:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.