Causal Estimation of User Learning in Personalized Systems

• URL: http://arxiv.org/abs/2306.00485v1
• Date: Thu, 1 Jun 2023 09:37:43 GMT
• Title: Causal Estimation of User Learning in Personalized Systems
• Abstract summary: We introduce a non-parametric causal model of user actions in a personalized system. We show that the Cookie-Cookie-Day experiment, designed for the measurement of the user learning effect, is biased when there is personalization. We derive new experimental designs that intervene in the personalization system to generate the variation necessary to separately identify the causal effect mediated through user learning and personalization.
• Score: 5.016998307223021
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In online platforms, the impact of a treatment on an observed outcome may change over time as 1) users learn about the intervention, and 2) the system personalization, such as individualized recommendations, change over time. We introduce a non-parametric causal model of user actions in a personalized system. We show that the Cookie-Cookie-Day (CCD) experiment, designed for the measurement of the user learning effect, is biased when there is personalization. We derive new experimental designs that intervene in the personalization system to generate the variation necessary to separately identify the causal effect mediated through user learning and personalization. Making parametric assumptions allows for the estimation of long-term causal effects based on medium-term experiments. In simulations, we show that our new designs successfully recover the dynamic causal effects of interest.

Related papers

• Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters [3.8028536988925477]
  We present three experiments that study the effects of interactive feedback collections on user impressions in domains with objective and subjective feedback.<n>Results show that in a context where there is an objectively correct answer, providing HITL feedback lowered both participants' trust in the system and their perception of system accuracy.<n>These results highlight the importance of considering the effects of allowing different types of end-user feedback on user trust when designing intelligent systems.
  arXiv  Detail & Related papers  (2026-07-20T04:46:25Z)
• BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces [53.55407626944686]
  textscBehaviorBench is a benchmark for evaluating personalized decision modeling from real-world behavioral traces.<n>textscBehaviorBench reconstructs wallet-level decision histories from observed public prediction-market and on-chain records.<n>Across 2,000 evaluation wallets, the benchmark contains 141,445 Belief instances and 1,485,972 Trade instances.
  arXiv  Detail & Related papers  (2026-06-01T19:04:36Z)
• Preference-Aware Rubric Learning for Personalized Evaluation [59.539429430690156]
  Existing evaluation methods fail to capture user-specific preferences embedded in long-term interaction histories.<n>We propose Personalized Evaluation as Learning, a paradigm that formulates personalized evaluation as a learning problem rather than a static judgment.<n>Experiments show that PARL consistently induces high-fidelity rubrics that reliably identify user-aligned responses and generalize across users.
  arXiv  Detail & Related papers  (2026-05-29T17:00:55Z)
• The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study [56.649987197786096]
  Large language models (LLMs) show potential as simulators of human behavior.<n>LLMs can induce unintended shifts in latent user attributes, causing user drift.<n>We formalize the confounding or selection bias that can arise due to user drift.
  arXiv  Detail & Related papers  (2026-05-20T06:09:41Z)
• Unobtrusive In-Situ Measurement of Behavior Change by Deep Metric Similarity Learning of Motion Patterns [25.896192890947216]
  This paper introduces an in-situ measurement method to detect user behavior changes during arbitrary exposures in XR systems.<n>We present a biometric user model based on deep metric similarity learning, which uses high-dimensional embeddings as reference vectors.
  arXiv  Detail & Related papers  (2025-09-04T12:46:18Z)
• Causal Sensitivity Identification using Generative Learning [25.118222387224677]
  We conduct causal impact analysis using interventional and counterfactual perspectives.<n>Our method exploits the Conditional Autoencoder (CVAE) to identify the causal impact and serve as a generative predictor.
  arXiv  Detail & Related papers  (2025-09-01T10:42:44Z)
• Algorithm Adaptation Bias in Recommendation System Online Experiments [4.8862630578310435]
  An underexplored but critical bias is algorithm adaptation effect.<n>Results often favor the production variant with large traffic while underestimating the performance of the test variant with small traffic.<n>We detail the mechanisms of this bias, present empirical evidence from real-world experiments, and discuss potential methods for a more robust online evaluation.
  arXiv  Detail & Related papers  (2025-08-29T19:23:04Z)
• Addressing Personalized Bias for Unbiased Learning to Rank [56.663619153713434]
  Unbiased learning to rank (ULTR) aims to learn unbiased ranking models from biased user behavior logs.<n>We propose a novel user-aware inverse-propensity-score estimator for learning-to-rank objectives.
  arXiv  Detail & Related papers  (2025-08-28T14:01:31Z)
• Personalized Counterfactual Framework: Generating Potential Outcomes from Wearable Data [1.7396556690675233]
  This paper introduces a framework to learn personalized counterfactual models from wearable data.<n>We first augment individual datasets with data from similar patients via multi-modal similarity analysis.<n>We then use a temporal PC (Peter-Clark) algorithm adaptation to discover predictive relationships.<n> Gradient Boosting Machines are trained on these relationships to quantify individual-specific effects.
  arXiv  Detail & Related papers  (2025-08-20T05:04:17Z)
• NextQuill: Causal Preference Modeling for Enhancing LLM Personalization [82.15961484963256]
  We introduce NextQuill, a novel personalization framework grounded in causal preference modeling.<n>Building on this insight, NextQuill introduces two complementary alignment strategies.<n> Experiments across multiple personalization benchmarks demonstrate that NextQuill significantly improves personalization quality.
  arXiv  Detail & Related papers  (2025-06-03T02:08:55Z)
• Exploring the Impact of Personality Traits on Conversational Recommender Systems: A Simulation with Large Language Models [70.180385882195]
  This paper introduces a personality-aware user simulation for Conversational Recommender Systems (CRSs) The user agent induces customizable personality traits and preferences, while the system agent possesses the persuasion capability to simulate realistic interaction in CRSs. Experimental results demonstrate that state-of-the-art LLMs can effectively generate diverse user responses aligned with specified personality traits.
  arXiv  Detail & Related papers  (2025-04-09T13:21:17Z)
• Accounting for Sycophancy in Language Model Uncertainty Estimation [28.08509288774144]
  We study the relationship between sycophancy and uncertainty estimation for the first time. We show that user confidence plays a critical role in modulating the effects of sycophancy. We argue that externalizing both model and user uncertainty can help to mitigate the impacts of sycophancy bias.
  arXiv  Detail & Related papers  (2024-10-17T18:00:25Z)
• Evaluating Alternative Training Interventions Using Personalized Computational Models of Learning [0.0]
  evaluating different training interventions to determine which produce the best learning outcomes is one of the main challenges faced by instructional designers. We present an approach for automatically tuning models to specific individuals and show that personalized models make better predictions of students' behavior than generic ones. Our approach makes predictions that align with previous human findings, as well as testable predictions that might be evaluated with future human experiments.
  arXiv  Detail & Related papers  (2024-08-24T22:51:57Z)
• Dual Test-time Training for Out-of-distribution Recommender System [91.15209066874694]
  We propose a novel Dual Test-Time-Training framework for OOD Recommendation, termed DT3OR. In DT3OR, we incorporate a model adaptation mechanism during the test-time phase to carefully update the recommendation model. To the best of our knowledge, this paper is the first work to address OOD recommendation via a test-time-training strategy.
  arXiv  Detail & Related papers  (2024-07-22T13:27:51Z)
• Causal Post-Processing of Predictive Models [1.3124513975412255]
  We propose causal post-processing, a family of techniques for refining predictive scores to better align with causal effects.<n>We introduce three main CPP approaches -- monotonic post-processing, correction post-processing, and model-based post-processing.<n>Our findings highlight the advantages of integrating non-causal predictive models with experimental data, rather than treating them as competing alternatives.
  arXiv  Detail & Related papers  (2024-06-13T20:18:16Z)
• POV Learning: Individual Alignment of Multimodal Models using Human Perception [1.4796543791607086]
  We argue that alignment on an individual level can boost the subjective predictive performance for the individual user interacting with the system. We test this, by integrating perception information into machine learning systems and measuring their predictive performance. Our findings suggest that exploiting individual perception signals for the machine learning of subjective human assessments provides a valuable cue for individual alignment.
  arXiv  Detail & Related papers  (2024-05-07T16:07:29Z)
• DOMINO: Visual Causal Reasoning with Time-Dependent Phenomena [59.291745595756346]
  We propose a set of visual analytics methods that allow humans to participate in the discovery of causal relations associated with windows of time delay. Specifically, we leverage a well-established method, logic-based causality, to enable analysts to test the significance of potential causes. Since an effect can be a cause of other effects, we allow users to aggregate different temporal cause-effect relations found with our method into a visual flow diagram.
  arXiv  Detail & Related papers  (2023-03-12T03:40:21Z)
• Zero-shot causal learning [64.9368337542558]
  CaML is a causal meta-learning framework which formulates the personalized prediction of each intervention's effect as a task. We show that CaML is able to predict the personalized effects of novel interventions that do not exist at the time of training.
  arXiv  Detail & Related papers  (2023-01-28T20:14:11Z)
• Fair Effect Attribution in Parallel Online Experiments [57.13281584606437]
  A/B tests serve the purpose of reliably identifying the effect of changes introduced in online services. It is common for online platforms to run a large number of simultaneous experiments by splitting incoming user traffic randomly. Despite a perfect randomization between different groups, simultaneous experiments can interact with each other and create a negative impact on average population outcomes.
  arXiv  Detail & Related papers  (2022-10-15T17:15:51Z)
• Personalizing Intervened Network for Long-tailed Sequential User Behavior Modeling [66.02953670238647]
  Tail users suffer from significantly lower-quality recommendation than the head users after joint training. A model trained on tail users separately still achieve inferior results due to limited data. We propose a novel approach that significantly improves the recommendation performance of the tail users.
  arXiv  Detail & Related papers  (2022-08-19T02:50:19Z)
• Learning Transferrable Parameters for Long-tailed Sequential User Behavior Modeling [70.64257515361972]
  We argue that focusing on tail users could bring more benefits and address the long tails issue. Specifically, we propose a gradient alignment and adopt an adversarial training scheme to facilitate knowledge transfer from the head to the tail.
  arXiv  Detail & Related papers  (2020-10-22T03:12:02Z)
• Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracy [8.11839312231511]
  Mixed-initiative systems allow users to interactively provide feedback to improve system performance. Our research investigates how the act of providing feedback can affect user understanding of an intelligent system and its accuracy.
  arXiv  Detail & Related papers  (2020-08-28T16:46:41Z)
• Differentially Private ERM Based on Data Perturbation [41.37436071802578]
  We measure the contributions of various training data instances on the final machine learning model. Considering that the key of our method is to measure each data instance separately, we propose a new Data perturbation' based (DB) paradigm for DP-ERM.
  arXiv  Detail & Related papers  (2020-02-20T06:05:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.