Impression Share Prediction: An Offline Evaluation Task for Ranking Systems

• URL: http://arxiv.org/abs/2608.16872v1
• Date: Mon, 17 Aug 2026 17:52:55 GMT
• Title: Impression Share Prediction: An Offline Evaluation Task for Ranking Systems
• Abstract summary: We propose emphimpression share prediction as an offline evaluation task.<n>The task is inherently counterfactual, since the candidate has never served live traffic.<n>We develop a statistical learning framework that predicts impression shares from a candidate's early-interaction confidence signals.
• Score: 2.455009405787869
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline evaluation is a major gateway before online evaluation of ranking models in A/B testing. Standard offline metrics measure predictive accuracy, but are only a surrogate for downstream utility: a model can improve them while redistributing impressions across objective buckets in ways that degrade downstream utility. No offline method surfaces these impression share shifts before online evaluation. We propose \emph{impression share prediction} as an offline evaluation task: given a candidate ranking model, predict the distribution of impressions it would produce across objective buckets - impressions grouped by optimization goal (e.g., click, video view). The task is inherently counterfactual, since the candidate has never served live traffic. We propose a structural causal model of how model predictions and delivery capacity jointly determine impression allocation, and show the counterfactual effect is identified from observational data. Building on this, we develop a statistical learning framework that predicts impression shares from a candidate's early-interaction confidence signals and current system state, trained on historical data. On data from multiple ranking model families, a Random Forest reduces L1 error by 49\% over a constant baseline for models seen during training. For held-out models, evaluated by time since first appearance, the first hour is the closest analog to true online evaluation and the hardest: the Random Forest falls below the baseline because the capacity state still reflects the prior model. An encoder-conditioned architecture that simulates a 2-hour rollout over recent auction dynamics recovers $+$22\% L1 in this regime.

Related papers

• Train-before-Test Harmonizes Language Model Rankings [39.02785699830391]
  Existing language model benchmarks provide contradictory model rankings, even for benchmarks that aim to capture similar skills.<n>We compare model potential by providing each model with identical benchmark-specific fine-tuning before evaluation.<n>We show that model potential rankings obtained through train-before-test exhibit remarkable consistency across all benchmarks.
  arXiv  Detail & Related papers  (2025-07-07T16:54:18Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Unified Human Localization and Trajectory Prediction with Monocular Vision [64.19384064365431]
  MonoTransmotion is a Transformer-based framework that uses only a monocular camera to jointly solve localization and prediction tasks.<n>We show that by jointly training both tasks with our unified framework, our method is more robust in real-world scenarios made of noisy inputs.
  arXiv  Detail & Related papers  (2025-03-05T14:18:39Z)
• Streaming egocentric action anticipation: An evaluation scheme and approach [27.391434284586985]
  Egocentric action anticipation aims to predict the future actions the camera wearer will perform from the observation of the past. Current evaluation schemes assume that predictions are available right after the input video is observed. We propose a streaming egocentric action evaluation scheme which assumes that predictions are performed online and made available only after the model has processed the current input segment.
  arXiv  Detail & Related papers  (2023-06-29T04:53:29Z)
• Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive? [93.10694819127608]
  We propose a unified evaluation pipeline for forecasting methods with real-world perception inputs. Our in-depth study uncovers a substantial performance gap when transitioning from curated to perception-based data.
  arXiv  Detail & Related papers  (2023-06-15T17:03:14Z)
• Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning [105.70602423944148]
  We propose a novel method, called value-consistent representation learning (VCR), to learn representations that are directly related to decision-making. Instead of aligning this imagined state with a real state returned by the environment, VCR applies a $Q$-value head on both states and obtains two distributions of action values. It has been demonstrated that our methods achieve new state-of-the-art performance for search-free RL algorithms.
  arXiv  Detail & Related papers  (2022-06-25T03:02:25Z)
• Selective Prediction via Training Dynamics [31.708701583736644]
  We show that state-of-the-art selective prediction performance can be attained solely from studying the training dynamics of a model.<n>In particular, we reject data points exhibiting too much disagreement with the final prediction at late stages in training.<n>The proposed rejection mechanism is domain-agnostic (i.e., it works for both discrete and real-valued prediction) and can be flexibly combined with existing selective prediction approaches.
  arXiv  Detail & Related papers  (2022-05-26T17:51:29Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• Towards Streaming Egocentric Action Anticipation [23.9991007631236]
  Egocentric action anticipation is the task of predicting the future actions a camera wearer will likely perform based on past video observations. Current evaluation schemes assume that predictions can be made offline, and hence that computational resources are not limited. We propose a streaming'' egocentric action anticipation evaluation protocol which explicitly considers model runtime for performance assessment.
  arXiv  Detail & Related papers  (2021-10-11T16:22:56Z)
• Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future [73.03458424369657]
  In real-time forecasting in public health, data collection is a non-trivial and demanding task. 'Backfill' phenomenon and its effect on model performance has been barely studied in the prior literature. We formulate a novel problem and neural framework Back2Future that aims to refine a given model's predictions in real-time.
  arXiv  Detail & Related papers  (2021-06-08T14:48:20Z)
• Modeling Online Behavior in Recommender Systems: The Importance of Temporal Context [30.894950420437926]
  We show how omitting temporal context when evaluating recommender system performance leads to false confidence. We propose a training procedure to further embed the temporal context in existing models. Results show that including our temporal objective can improve recall@20 by up to 20%.
  arXiv  Detail & Related papers  (2020-09-19T19:36:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.