Learning Next Action Predictors from Human-Computer Interaction

• URL: http://arxiv.org/abs/2603.05923v1
• Date: Fri, 06 Mar 2026 05:31:04 GMT
• Title: Learning Next Action Predictors from Human-Computer Interaction
• Abstract summary: LongNAP is a user model that combines parametric and in-context learning to reason over long interaction histories.<n>We label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time.<n>LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data.
• Score: 53.41504288073184
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Truly proactive AI systems must anticipate what we will do next. This foresight demands far richer information than the sparse signals we type into our prompts -- it demands reasoning over the entire context of what we see and do. We formalize this as next action prediction (NAP): given a sequence of a user's multimodal interactions with a computer (screenshots, clicks, sensor data), predict that user's next action. Progress on this task requires both new data and modeling approaches. To scale data, we annotate longitudinal, naturalistic computer use with vision-language models. We release an open-source pipeline for performing this labeling on private infrastructure, and label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time. We then introduce LongNAP, a user model that combines parametric and in-context learning to reason over long interaction histories. LongNAP is trained via policy gradient methods to generate user-specific reasoning traces given some context; retrieve relevant traces from a library of past traces; and then apply retrieved traces in-context to predict future actions. Using an LLM-as-judge evaluation metric (0-1 similarity to ground truth), LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data (by 79% and 39% respectively). Additionally, LongNAP generalizes to held out users when trained across individuals. The space of next actions a user might take at any moment is unbounded, spanning thousands of possible outcomes. Despite this, 17.1% of LongNAP's predicted trajectories are well-aligned with what a user does next (LLM-judge score $\geq$ 0.5). This rises to 26% when we filter to highly confident predictions. In sum, we argue that learning from the full context of user behavior to anticipate user needs is now a viable task with substantial opportunity.

Related papers

• OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs [25.570460595046058]
  Recent omni-modal large language models ( Omni-LLMs) show great potential as real-time video assistants.<n>Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help.
  arXiv  Detail & Related papers  (2026-08-21T17:59:52Z)
• Action-Aware Generative Sequence Modeling for Short Video Recommendation [50.74467504063892]
  Short videos often contain diverse segments, and users may not hold the same attitude toward all of them.<n>Traditional binary-classification recommendation models, which treat a video as a single holistic entity, face limitations in accurately capturing such nuanced preferences.<n>We propose a novel modeling paradigm: Action-Aware Generative Sequence Network (A2Gen), which refines user actions along the temporal dimension.
  arXiv  Detail & Related papers  (2026-04-28T16:41:04Z)
• Towards Realistic Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions [50.70965714314064]
  Large Language Models (LLMs) are increasingly serving as personal assistants, where users share complex and diverse preferences over extended interactions.<n>This work proposes RealPref, a benchmark for evaluating realistic preference-following in personalized user-LLM interactions.
  arXiv  Detail & Related papers  (2026-03-04T15:42:43Z)
• Mind the Gap! Static and Interactive Evaluations of Large Audio Models [55.87220295533817]
  Large Audio Models (LAMs) are designed to power voice-native experiences.<n>This study introduces an interactive approach to evaluate LAMs and collect 7,500 LAM interactions from 484 participants.
  arXiv  Detail & Related papers  (2025-02-21T20:29:02Z)
• New User Event Prediction Through the Lens of Causal Inference [20.676353189313737]
  We propose a novel discrete event prediction framework for new users with limited history.<n>We treat the user event history as the "treatment" for future events and the user category as the key confounder.<n>We demonstrate the improved performance of the proposed framework with a numerical simulation study and two real-world applications.
  arXiv  Detail & Related papers  (2024-07-08T05:35:54Z)
• Look into the Future: Deep Contextualized Sequential Recommendation [28.726897673576865]
  We propose a novel framework of sequential recommendation called Look into the Future (LIFT) LIFT builds and leverages the contexts of sequential recommendation. In our experiments, LIFT achieves significant performance improvement on click-through rate prediction and rating prediction tasks.
  arXiv  Detail & Related papers  (2024-05-23T09:34:28Z)
• USE: Dynamic User Modeling with Stateful Sequence Models [26.74966828348815]
  User Stateful Embedding (USE) generates user embeddings without the need for exhaustive reprocessing. We introduce a novel training objective named future W-behavior prediction to transcend the limitations of next-token prediction. We conduct experiments on 8 downstream tasks using Snapchat users' behavioral logs in both static (i.e., fixed user behavior sequences) and dynamic (i.e. periodically updated user behavior sequences) settings.
  arXiv  Detail & Related papers  (2024-03-20T07:05:19Z)
• A Nonparametric Bayes Approach to Online Activity Prediction [11.934335703226404]
  We propose a novel approach to predict the number of users that will be active in a given time period. We derive closed-form expressions for the number of new users expected in a given period, and a simple Monte Carlo algorithm targeting the posterior distribution of the number of days needed to attain a desired number of users.
  arXiv  Detail & Related papers  (2024-01-26T09:11:42Z)
• Latent User Intent Modeling for Sequential Recommenders [92.66888409973495]
  Sequential recommender models learn to predict the next items a user is likely to interact with based on his/her interaction history on the platform. Most sequential recommenders however lack a higher-level understanding of user intents, which often drive user behaviors online. Intent modeling is thus critical for understanding users and optimizing long-term user experience.
  arXiv  Detail & Related papers  (2022-11-17T19:00:24Z)
• PinnerFormer: Sequence Modeling for User Representation at Pinterest [60.335384724891746]
  We introduce PinnerFormer, a user representation trained to predict a user's future long-term engagement. Unlike prior approaches, we adapt our modeling to a batch infrastructure via our new dense all-action loss. We show that by doing so, we significantly close the gap between batch user embeddings that are generated once a day and realtime user embeddings generated whenever a user takes an action.
  arXiv  Detail & Related papers  (2022-05-09T18:26:51Z)
• Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction [57.56466850377598]
  Reasoning over visual data is a desirable capability for robotics and vision-based applications. In this paper, we present a framework on graph to uncover relationships in different objects in the scene for reasoning about pedestrian intent. Pedestrian intent, defined as the future action of crossing or not-crossing the street, is a very crucial piece of information for autonomous vehicles.
  arXiv  Detail & Related papers  (2020-02-20T18:50:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.