Towards a Densing Law for User Representation Learning at Billion-Scale Capacity

• URL: http://arxiv.org/abs/2608.23392v1
• Date: Mon, 24 Aug 2026 15:38:07 GMT
• Title: Towards a Densing Law for User Representation Learning at Billion-Scale Capacity
• Abstract summary: We propose User Behavioral Densing Law for characterizing the relationship between data scale and the minimum sufficient tokenization capacity.<n>We find an approximately linear relationship between the logarithms of minimum sufficient tokenization capacity and input data size measured by tokens.<n>We further develop ALGN, an adaptive variable-length tokenization method that improves capacity allocation.
• Score: 23.385857270290032
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: User representation learning in real-world industrial scenarios is commonly scaled by increasing user amount, behavioral sequence length and model size. However, existing methods face two challenges: (i) Bottleneck for raw data scaling at billion-scale capacity, as performance exhibit diminishing performance gains with larger-scale raw text user behavioral input, which can be mitigated by tokenization. (ii) Lack of quantitative analysis of how tokenization configurations should scale with data size. In this report, we propose User Behavioral Densing Law for characterizing the quantitative relationship between data scale and the minimum sufficient tokenization capacity. Firstly, we conduct a pilot study on raw & tokenized scaling comparison on billion-scale Alipay dataset, revealing the raw data scaling bottleneck and the sustained gains enabled by tokenization. To derive the scaling pattern governing the minimum sufficient tokenization configuration at different data scales, theoretical analysis and systematic experiments are employed to summarize the quantitative scaling pattern. We find an approximately linear relationship between the logarithms of minimum sufficient tokenization capacity and input data size measured by tokens, and the scaling slope varies systematically with the tokenization method and data source, reflecting differences in representation-space redundancy and intra-source uniqueness. Guided by the proposed law, we further develop ALGN, an adaptive variable-length tokenization method that improves capacity allocation. Extensive experiments across diverse data sources, tokenization methods, and downstream tasks demonstrate the generalizability and reliability of the User Behavioral Densing Law, providing practical guidance for tokenization configuration selection in large-scale user representation learning. Moreover, ALGN outperforms existing baselines.

Related papers

• Data-Efficient Deep Learning: Empirical Guidelines for Training Set Size Estimation in Inertial Sensor Classification [0.0]
  We introduce a unified framework that analyzes classification performance under both binary and multi-class scenarios.<n>We propose a quantitative stability point metric, defined as the sample size required for the learning curve to stabilize.<n>We offer a generalizable framework to extrapolate total data requirements from small-scale pilot studies.
  arXiv  Detail & Related papers  (2026-07-10T13:30:28Z)
• Scaling Laws for Reranking in Information Retrieval [24.00475965133032]
  We present the first systematic study of scaling laws for rerankers.<n>Using a detailed case study with cross-encoder rerankers, we demonstrate that performance follows a predictable power law.<n>Our results establish scaling principles for reranking and provide actionable insights for building industrial-grade retrieval systems.
  arXiv  Detail & Related papers  (2026-03-05T05:03:07Z)
• Scaling Laws for Embedding Dimension in Information Retrieval [26.21690287784803]
  We conduct a comprehensive analysis of the relationship between embedding dimension and retrieval performance.<n>We find that the scaling behavior fits a power law, allowing us to derive scaling laws for performance given only embedding dimension.<n>Our analysis shows that for evaluation tasks aligned with the training task, performance continues to improve as embedding size increases.
  arXiv  Detail & Related papers  (2026-02-04T21:27:12Z)
• Accelerate Speculative Decoding with Sparse Computation in Verification [49.74839681322316]
  Speculative decoding accelerates autoregressive language model inference by verifying multiple draft tokens in parallel.<n>Existing sparsification methods are designed primarily for standard token-by-token autoregressive decoding.<n>We propose a sparse verification framework that jointly sparsifies attention, FFN, and MoE components during the verification stage to reduce the dominant computation cost.
  arXiv  Detail & Related papers  (2025-12-26T07:53:41Z)
• Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks [100.13335639780415]
  Scaling laws often follow the power-law and proposed several variants of power-law functions to predict the scaling behavior at larger scales.<n>Existing methods mostly rely on point estimation and do not quantify uncertainty, which is crucial for real-world applications.<n>In this work, we explore a Bayesian framework based on Prior-data Fitted Networks (PFNs) for neural scaling law extrapolation.
  arXiv  Detail & Related papers  (2025-05-29T03:19:17Z)
• Compute Optimal Scaling of Skills: Knowledge vs Reasoning [50.76705503978189]
  We ask whether compute-optimal scaling behaviour can be skill-dependent.<n>In particular, we examine knowledge and reasoning-based skills such as knowledge-based QA and code generation.<n>We conclude with an analysis of how our findings relate to standard compute-optimal scaling using a validation set.
  arXiv  Detail & Related papers  (2025-03-13T05:21:22Z)
• Scaling laws in wearable human activity recognition [19.49701947129144]
  Scaling laws have the potential to help move towards more principled design by linking model capacity with pre-training data volume.<n>By conducting an exhaustive grid search on both amount of pre-training data and Transformer architectures, we establish the first known scaling laws for HAR.<n>We show that these scaling laws translate to downstream performance improvements on three HAR benchmark datasets.
  arXiv  Detail & Related papers  (2025-02-05T17:00:08Z)
• Scaling Laws for the Value of Individual Data Points in Machine Learning [55.596413470429475]
  We introduce a new perspective by investigating scaling behavior for the value of individual data points. We provide learning theory to support our scaling law, and we observe empirically that it holds across diverse model classes. Our work represents a first step towards understanding and utilizing scaling properties for the value of individual data points.
  arXiv  Detail & Related papers  (2024-05-30T20:10:24Z)
• Scaling Laws For Dense Retrieval [22.76001461620846]
  We investigate whether the performance of dense retrieval models follows the scaling law as other neural models. Results indicate that, under our settings, the performance of dense retrieval models follows a precise power-law scaling related to the model size and the number of annotations.
  arXiv  Detail & Related papers  (2024-03-27T15:27:36Z)
• Scaling Laws for Sparsely-Connected Foundation Models [70.41266138010657]
  We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets. We identify the first scaling law describing the relationship between weight sparsity, number of non-zero parameters, and amount of training data.
  arXiv  Detail & Related papers  (2023-09-15T16:29:27Z)
• A Solvable Model of Neural Scaling Laws [72.8349503901712]
  Large language models with a huge number of parameters, when trained on near internet-sized number of tokens, have been empirically shown to obey neural scaling laws. We propose a statistical model -- a joint generative data model and random feature model -- that captures this neural scaling phenomenology. Key findings are the manner in which the power laws that occur in the statistics of natural datasets are extended by nonlinear random feature maps.
  arXiv  Detail & Related papers  (2022-10-30T15:13:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.