Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation

• URL: http://arxiv.org/abs/2604.09368v1
• Date: Fri, 10 Apr 2026 14:38:45 GMT
• Title: Through Their Eyes: Fixation-aligned Tuning for Personalized User Emulation
• Abstract summary: We investigate whether aligning a vision-language model's (VLM's) visual attention with user-specific gaze patterns can improve simulation fidelity.<n>Our approach first probes the VLM's internal visual attention via interpretability operators to obtain a slot-level relevance distribution.<n>Experiments across three interpretability-based probing operators and two architecturally distinct VLM backbones demonstrate consistent improvements in both attention alignment and click prediction accuracy.
• Score: 11.879346281714453
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language model (LLM) agents are increasingly deployed as scalable user simulators for recommender system evaluation. Yet existing simulators perceive recommendations through text or structured metadata rather than the visual interfaces real users browse-a critical gap, since attention over recommendation layouts is both visually driven and highly personalized. We investigate whether aligning a vision-language model's (VLM's) visual attention with user-specific gaze patterns can improve simulation fidelity. Analysis of a real-world eye-tracking dataset collected in a carousel-based recommendation setting reveals that users exhibit stable individual gaze patterns strongly predictive of click behavior. Building on this finding, we propose Fixation-Aligned Tuning for user Emulation (FixATE). Our approach first probes the VLM's internal visual attention via interpretability operators to obtain a slot-level relevance distribution comparable with human fixation, and then learns personalized soft prompts to steer the model's attention toward each user's characteristic fixation pattern. Experiments across three interpretability-based probing operators and two architecturally distinct VLM backbones demonstrate consistent improvements in both attention alignment and click prediction accuracy. These results suggest that making the model "see like the user" is a viable path toward simulators that more faithfully reproduce how users perceive and act in recommendation interfaces.

Related papers

• From Overlooked to Explored: Recovering Item Relations via Mixture of Perspectives for Sequential Recommendation [18.117610268256005]
  We show that self-attention is dominant in sequential recommendation for its ability to compute pairwise item interactions.<n>We propose PRISM, a module that re-examines item relations from multiple perspectives.<n>PRISM consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-08-12T09:39:04Z)
• Personal Visual Context Learning in Large Multimodal Models [53.33351452841322]
  We present Personal Visual Context Learning (Personal ), the prompt-time capability of using user-specific visual context to resolve personalized queries.<n>Our analysis of LMMs identifies a profound context utilization gap, revealing that the mechanisms for leveraging visual evidence, as well as aggregating multiple visual observations, remain critically understudied.<n>Motivated by these findings, we propose the Agentic Context Bank, a strong-time baseline that structures a user's visual context into a self-refining memory bank and employs query-adaptive evidence selection.
  arXiv  Detail & Related papers  (2026-05-11T17:59:17Z)
• ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders [48.83868690303791]
  We introduce ConvApparel, a new dataset of human-AI conversations designed to address this gap.<n>Its unique dual-agent data collection protocol -- using both "good" and "bad" recommenders -- enables counterfactual validation.<n>We propose a comprehensive validation framework that combines statistical alignment, a human-likeness score, and counterfactual validation.
  arXiv  Detail & Related papers  (2026-02-18T23:00:21Z)
• Synthetic Interaction Data for Scalable Personalization in Large Language Models [67.31884245564086]
  We introduce a high-fidelity synthetic data generation framework called PersonaGym.<n>Unlike prior work that treats personalization as static persona-preference pairs, PersonaGym models a dynamic preference process.<n>We release PersonaAtlas, a large-scale, high-quality, and diverse synthetic dataset of high-fidelity multi-turn personalized interaction trajectories.
  arXiv  Detail & Related papers  (2026-02-12T20:41:22Z)
• Resolving Ambiguity in Gaze-Facilitated Visual Assistant Interaction Paradigm [36.752693539572086]
  We introduce GLARIFY, a novel method to leverage gaze information to enhance the model's effectiveness in real-world applications.<n>We analyzed hundreds of samples with the gaze modality to demonstrate the noisy nature of users' gaze patterns.<n>Experiments demonstrate that GLARIFY significantly outperforms baselines.
  arXiv  Detail & Related papers  (2025-09-26T07:02:40Z)
• Conv4Rec: A 1-by-1 Convolutional AutoEncoder for User Profiling through Joint Analysis of Implicit and Explicit Feedbacks [35.7275102787435]
  We introduce a new convolutional AutoEncoder architecture for user modelling and recommendation tasks.<n>Our model is able to learn jointly from both the explicit ratings and the implicit information in the sampling pattern.<n>In experiments on several real-life datasets, we achieve state-of-the-art performance on both the implicit and explicit feedback prediction tasks.
  arXiv  Detail & Related papers  (2025-09-09T08:25:11Z)
• Multimodal Foundation Model-Driven User Interest Modeling and Behavior Analysis on Short Video Platforms [4.393914222141582]
  This paper proposes a multimodal foundation model-based framework for user interest modeling and behavior analysis.<n>We introduce a behavior-driven feature embedding mechanism that incorporates viewing, liking, and commenting sequences to model dynamic interest evolution.<n>Results demonstrate significant improvements in behavior prediction accuracy, interest modeling for cold-start users, and recommendation click-through rates.
  arXiv  Detail & Related papers  (2025-09-05T02:05:10Z)
• SymCERE: Symmetric Contrastive Learning for Robust Review-Enhanced Recommendation [2.087411180679868]
  We propose SymCERE, a contrastive learning method that addresses false negatives in recommendation.<n>Experiments on 15 datasets from three platforms demonstrate that SymCERE outperforms several strong baselines.
  arXiv  Detail & Related papers  (2025-04-03T00:40:09Z)
• Interactive Visualization Recommendation with Hier-SUCB [52.11209329270573]
  We propose an interactive personalized visualization recommendation (PVisRec) system that learns on user feedback from previous interactions.<n>For more interactive and accurate recommendations, we propose Hier-SUCB, a contextual semi-bandit in the PVisRec setting.
  arXiv  Detail & Related papers  (2025-02-05T17:14:45Z)
• Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement [102.22911097049953]
  Large vision-language models (LVLMs) have achieved impressive results in visual question-answering and reasoning tasks.<n>Existing methods often depend on external models or data, leading to uncontrollable and unstable alignment results.<n>We propose SIMA, a self-improvement framework that enhances visual and language modality alignment without external dependencies.
  arXiv  Detail & Related papers  (2024-05-24T23:09:27Z)
• Latent User Intent Modeling for Sequential Recommenders [92.66888409973495]
  Sequential recommender models learn to predict the next items a user is likely to interact with based on his/her interaction history on the platform. Most sequential recommenders however lack a higher-level understanding of user intents, which often drive user behaviors online. Intent modeling is thus critical for understanding users and optimizing long-term user experience.
  arXiv  Detail & Related papers  (2022-11-17T19:00:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.