Text-to-Distribution Prediction with Quantile Tokens and Neighbor Context

• URL: http://arxiv.org/abs/2604.20216v1
• Date: Wed, 22 Apr 2026 06:05:22 GMT
• Title: Text-to-Distribution Prediction with Quantile Tokens and Neighbor Context
• Abstract summary: We study distributional regression under empirical-quantile supervision, where each input is paired with multiple observed quantile outcomes, and the target distribution is represented by a dense grid of quantiles.<n>We introduce Quantile Token Regression, which, to our knowledge, is the first work to insert dedicated quantile tokens into the input sequence, enabling direct input-output pathways for each quantile through self-attention.
• Score: 29.26696177839968
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many applications of LLM-based text regression require predicting a full conditional distribution rather than a single point value. We study distributional regression under empirical-quantile supervision, where each input is paired with multiple observed quantile outcomes, and the target distribution is represented by a dense grid of quantiles. We address two key limitations of current approaches: the lack of local grounding for distribution estimates, and the reliance on shared representations that create an indirect bottleneck between inputs and quantile outputs. In this paper, we introduce Quantile Token Regression, which, to our knowledge, is the first work to insert dedicated quantile tokens into the input sequence, enabling direct input-output pathways for each quantile through self-attention. We further augment these quantile tokens with retrieval, incorporating semantically similar neighbor instances and their empirical distributions to ground predictions with local evidence from similar instances. We also provide the first theoretical analysis of loss functions for quantile regression, clarifying which distributional objectives each optimizes. Experiments on the Inside Airbnb and StackSample benchmark datasets with LLMs ranging from 1.7B to 14B parameters show that quantile tokens with neighbors consistently outperform baselines (~4 points lower MAPE and 2x narrower prediction intervals), with especially large gains on smaller and more challenging datasets where quantile tokens produce substantially sharper and more accurate distributions.

Related papers

• Learn from your own latents and not from tokens: A sample-complexity theory [53.5821824211418]
  We study networks trained to predict their emphown latent representations of related views or masked regions.<n>We show that latent prediction achieves this with a number of samples emphconstant in $L$, up to logarithmic factors.<n>This suggests that explicit stacking such as H-JEPA is largely redundant.
  arXiv  Detail & Related papers  (2026-05-26T22:16:42Z)
• Posterior Augmented Flow Matching [64.1559809786948]
  Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
  arXiv  Detail & Related papers  (2026-05-01T17:59:59Z)
• Evidence-based Distributional Alignment for Large Language Models [58.65469623911573]
  We propose Evi-DA, an evidence-based alignment technique that improves the fidelity and robustness of LLM-based distribution estimation.<n>Given a target country, Evi-DA retrieves related World Values Survey items and their answer distributions, predicts a coarse Welzel value signature for each option, and infers the country-conditioned answer distribution in a structured format.
  arXiv  Detail & Related papers  (2026-03-03T03:34:06Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs)<n>We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length.<n>PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• Semiparametric conformal prediction [79.6147286161434]
  We construct a conformal prediction set accounting for the joint correlation structure of the vector-valued non-conformity scores.<n>We flexibly estimate the joint cumulative distribution function (CDF) of the scores.<n>Our method yields desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• Probabilistic Contrastive Learning for Long-Tailed Visual Recognition [78.70453964041718]
  Longtailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. We propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space.
  arXiv  Detail & Related papers  (2024-03-11T13:44:49Z)
• Minimally Supervised Learning using Topological Projections in Self-Organizing Maps [55.31182147885694]
  We introduce a semi-supervised learning approach based on topological projections in self-organizing maps (SOMs) Our proposed method first trains SOMs on unlabeled data and then a minimal number of available labeled data points are assigned to key best matching units (BMU) Our results indicate that the proposed minimally supervised model significantly outperforms traditional regression techniques.
  arXiv  Detail & Related papers  (2024-01-12T22:51:48Z)
• Multiple Hypothesis Dropout: Estimating the Parameters of Multi-Modal Output Distributions [22.431244647796582]
  This paper presents a Mixture of Multiple-Output functions (MoM) approach using a novel variant of dropout, Multiple Hypothesis Dropout. Experiments on supervised learning problems illustrate that our approach outperforms existing solutions for reconstructing multimodal output distributions. Additional studies on unsupervised learning problems show that estimating the parameters of latent posterior distributions within a discrete autoencoder significantly improves codebook efficiency, sample quality, precision and recall.
  arXiv  Detail & Related papers  (2023-12-18T22:20:11Z)
• Neural Spline Search for Quantile Probabilistic Modeling [35.914279831992964]
  We propose a non-parametric and data-driven approach, Neural Spline Search (NSS), to represent the observed data distribution without parametric assumptions. We demonstrate that NSS outperforms previous methods on synthetic, real-world regression and time-series forecasting tasks.
  arXiv  Detail & Related papers  (2023-01-12T07:45:28Z)
• Auxiliary Quantile Forecasting with Linear Networks [6.155158115218501]
  We propose a novel multi-task method for quantile forecasting with shared Linear layers. Our method is based on the Implicit quantile learning approach. We show learning auxiliary quantile tasks leads to state-of-the-art performance on deterministic forecasting benchmarks.
  arXiv  Detail & Related papers  (2022-12-05T20:09:32Z)
• Learning Quantile Functions without Quantile Crossing for Distribution-free Time Series Forecasting [12.269597033369557]
  We propose the Incremental (Spline) Quantile Functions I(S)QF, a flexible and efficient distribution-free quantile estimation framework. We also provide a generalization error analysis of our proposed approaches under the sequence-to-sequence setting.
  arXiv  Detail & Related papers  (2021-11-12T06:54:48Z)
• Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning [14.967168108174466]
  We first propose a non-decreasing quantile function network (NDQFN) to guarantee the monotonicity of the obtained quantile estimates. We then design a general exploration framework called distributional prediction error (DPE) which utilizes the entire distribution of the quantile function.
  arXiv  Detail & Related papers  (2021-05-14T08:12:51Z)
• Regularization Strategies for Quantile Regression [8.232258589877942]
  We show that minimizing an expected pinball loss over a continuous distribution of quantiles is a good regularizer even when only predicting a specific quantile. We show that lattice models enable regularizing the predicted distribution to a location-scale family.
  arXiv  Detail & Related papers  (2021-02-09T21:10:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.