SERUM: State Extraction and Refinement for User Modeling

• URL: http://arxiv.org/abs/2607.29181v1
• Date: Fri, 31 Jul 2026 09:02:33 GMT
• Title: SERUM: State Extraction and Refinement for User Modeling
• Abstract summary: We present SERUM, a framework that extracts finite-state behavioral models directly from unstructured egocentric video.<n>We evaluate behavioral structure by fitting first-order Markov models over the resulting label sequences.<n>Across 61 egocentric videos in four domains, we find: (1) iterative label refinement converges to a stable state vocabulary, which we term equilibrium, after several passes; (2) normalized Markov models achieve substantially lower perplexity and higher action predictions than frequency baselines; and (3) human annotators rate final-pass labels as accurate and meaningfully improved over first-pass labels.
• Score: 17.94894550460533
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic assistants capable of proactive, personalized interactions require structured models of user intent and workflow. However, building these models from raw, unstructured screen activity remains an open challenge. We present SERUM, a multi-pass framework that extracts finite-state behavioral models directly from unstructured egocentric video using hierarchical VLM annotation. Processing screen recordings through a sliding window, SERUM alternates between activity-recognition and intent-inference passes, with each pass refining labels using accumulated prior context to reduce hallucination and temporal conflation seen in single-pass annotation. Synonymous states are then merged via sentence embeddings and human-calibrated thresholds into a compact, coherent taxonomy. We evaluate behavioral structure by fitting first-order Markov models over the resulting label sequences (both actions and intents) and measuring predictive accuracy against frequency baselines. Across 61 egocentric videos in four domains (coding, cooking, physical activities, and daily life), we find: (1) iterative label refinement converges to a stable state vocabulary, which we term schematic equilibrium, after several passes; (2) normalized Markov models achieve substantially lower perplexity and higher action predictions than frequency baselines, with the largest gains on structured tasks like coding; and (3) human annotators rate final-pass labels as accurate and meaningfully improved over first-pass labels. To our knowledge, SERUM is the first system to produce interpretable process models from unstructured egocentric screen video without manual annotation, opening a scalable pathway for user modeling and behavioral understanding in the wild. Our demo, code, and results are publicly available

Related papers

• Compositional Benchmark Synthesis for Hierarchical Human Action Recognition [4.812022454180755]
  A benchmark-generation and evaluation frameworkis proposed that synthesizes a four-level hierarchical-intention benchmark.<n> Episodes are assembled by a transition model under a subject-consistency constraint.<n>A coverage-aware sampler reduces the subject usage Gini from 0.566 to 0.248.
  arXiv  Detail & Related papers  (2026-08-11T10:22:22Z)
• DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction [21.818459524136376]
  DeepGaze3.5-VL establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of Information Gain (IG) on MIT1003, a 46% improvement over DeepGaze III.<n>Our generative framework serves as a powerful computational tool for direct behavioral interventions, allowing for controlled in-silico simulations that would be experimentally difficult or impossible to conduct in vivo.
  arXiv  Detail & Related papers  (2026-07-02T12:25:45Z)
• DisPOSE: Projected Polystochastic Diffusion for Self-Supervised Multi-View 3D Human Pose Estimation [58.47973015036709]
  DisPOSE is a self-supervised framework that approximates the inherently discrete multi-view person-assignment problem.<n>By employing differentiable Sinkhorn projections during denoising, our model learns to guide solutions toward valid and feasible assignments.<n>The proposed approach outperforms current state-of-the-art self-supervised methods on standard datasets.
  arXiv  Detail & Related papers  (2026-06-05T16:14:39Z)
• SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing [76.349958946335]
  We present SAMA (factorized Semantic Anchoring and Motion Alignment), a framework that factorizes video editing into semantic anchoring and motion modeling.<n>First, we introduce Semantic Anchoring, which establishes a reliable visual anchor by jointly predicting semantic tokens and video latents at sparse anchor frames.<n>Second, Motion Alignment pre-trains the same backbone on motion-centric video restoration pretext tasks.
  arXiv  Detail & Related papers  (2026-03-19T17:59:51Z)
• Peeking Into The Future For Contextual Biasing [8.769657210925777]
  We propose a contextual biasing method for attention based encoder decoder (AED) models.<n>We simultaneously predict multiple future tokens, enabling the model to "peek into the future" and score potential candidate entities.<n>Our approach achieves up to 50.34% relative improvement in named entity word error rate compared to the baseline AED model.
  arXiv  Detail & Related papers  (2025-12-19T14:56:28Z)
• Bridging Weakly-Supervised Learning and VLM Distillation: Noisy Partial Label Learning for Efficient Downstream Adaptation [51.67328507400985]
  In noisy partial label learning (NPLL), each training sample is associated with a set of candidate labels annotated by multiple noisy annotators.<n>This paper focuses on learning from partial labels annotated by pre-trained vision-language models.<n>It proposes an innovative collaborative consistency regularization (Co-Reg) method.
  arXiv  Detail & Related papers  (2025-06-03T12:48:54Z)
• Higher-Order DeepTrails: Unified Approach to *Trails [7.270112855088838]
  Analyzing, understanding, and describing human behavior is advantageous in different settings, such as web browsing or traffic navigation. We propose to analyze entire sequences using autoregressive language models, as they are traditionally used to model higher-order dependencies in sequences. We show that our approach can be easily adapted to model different settings introduced in previous work, namely HypTrails, MixedTrails and even SubTrails.
  arXiv  Detail & Related papers  (2023-10-06T06:54:11Z)
• STMT: A Spatial-Temporal Mesh Transformer for MoCap-Based Action Recognition [50.064502884594376]
  We study the problem of human action recognition using motion capture (MoCap) sequences. We propose a novel Spatial-Temporal Mesh Transformer (STMT) to directly model the mesh sequences. The proposed method achieves state-of-the-art performance compared to skeleton-based and point-cloud-based models.
  arXiv  Detail & Related papers  (2023-03-31T16:19:27Z)
• LESS: Label-Efficient Semantic Segmentation for LiDAR Point Clouds [62.49198183539889]
  We propose a label-efficient semantic segmentation pipeline for outdoor scenes with LiDAR point clouds. Our method co-designs an efficient labeling process with semi/weakly supervised learning. Our proposed method is even highly competitive compared to the fully supervised counterpart with 100% labels.
  arXiv  Detail & Related papers  (2022-10-14T19:13:36Z)
• An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild [27.943550651941166]
  We tackle the task of video-based audio-visual emotion recognition, within the premises of the 2nd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW) Standard methodologies that rely solely on the extraction of facial features often fall short of accurate emotion prediction in cases where the aforementioned source of affective information is inaccessible due to head/body orientation, low resolution and poor illumination. We aspire to alleviate this problem by leveraging bodily as well as contextual features, as part of a broader emotion recognition framework.
  arXiv  Detail & Related papers  (2021-07-07T20:13:17Z)
• Document Ranking with a Pretrained Sequence-to-Sequence Model [56.44269917346376]
  We show how a sequence-to-sequence model can be trained to generate relevance labels as "target words" Our approach significantly outperforms an encoder-only model in a data-poor regime.
  arXiv  Detail & Related papers  (2020-03-14T22:29:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.