Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users

• URL: http://arxiv.org/abs/2606.20482v1
• Date: Thu, 18 Jun 2026 17:00:48 GMT
• Title: Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users
• Abstract summary: Most existing methods collect explicit human feedback and train a reward model to predict the human preference based on the response text.<n>We build a new dataset called IFLLM, which collects 1336 multi-turn questions from the 59 Mechanical Turk workers.<n>Our reward model based on the implicit user feedback boosts the accuracy of the text-based reward model from 55% to 64% and nearly triples the relative response quality improvements.
• Score: 25.631351524871636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: To align a Large Language Model (LLM), most existing methods collect explicit human feedback and train a reward model to predict the human preference based on the response text. These existing methods have two key limitations. First, the users rarely provide explicit feedback for LLM responses, which makes the high-quality preference annotation expensive to collect. Second, the methods do not leverage implicit human feedback, which has proven vital to the economic moats of Internet giants. To quantify the value of implicit feedback, we build a new dataset called IFLLM, which collects 1336 multi-turn questions from the 59 Mechanical Turk workers, their mouse trajectories, and eye gazing points to the LLMs' responses from their webcams. IFLLM shows that the users have very diverse types of gazing behavior and mouse trajectories. Our reward model based on the implicit user feedback boosts the accuracy of the text-based reward model from 55% to 64% and nearly triples the relative response quality improvements after applying the DPO to eight LLMs, demonstrating the value of implicit feedback in the wild. Our data collection website, dataset, and codes can be found at https://github.com/themehulpatwari/llm-implicit-feedback/.

Related papers

• Interactive Episodic Memory with User Feedback [14.503449526387776]
  We introduce the Episodic Memory with Questions and Feedback task (EM-QnF)<n>Here, the user can provide feedback on the model's initial prediction or add more information.<n>We propose a lightweight training scheme that avoids expensive sequential optimization.
  arXiv  Detail & Related papers  (2026-04-27T18:19:58Z)
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards [88.82702433508393]
  We propose treating verbal feedback as a conditioning signal.<n>Inspired by language priors in text-to-image generation, we introduce the feedback-conditional policy.
  arXiv  Detail & Related papers  (2025-09-26T17:58:27Z)
• Multimodal LLMs as Customized Reward Models for Text-to-Image Generation [60.164968941945645]
  We introduce LLaVA-Reward, an efficient reward model designed to automatically evaluate text-to-image (T2I) generations across multiple perspectives.<n>LLaVA-Reward directly utilizes the hidden states of multimodal large language models (MLLMs)<n>We train LLaVA-Reward on four evaluation perspectives: text-image alignment, fidelity/artifact, safety, and overall ranking.
  arXiv  Detail & Related papers  (2025-07-28T23:52:53Z)
• Prompt-Based LLMs for Position Bias-Aware Reranking in Personalized Recommendations [0.0]
  Large language models (LLMs) have been adopted for prompt-based recommendation.<n>LLMs face limitations such as limited context window size, inefficient pointwise and pairwise prompting, and difficulty handling listwise ranking.<n>We propose a hybrid framework that combines a traditional recommendation model with an LLM for reranking top-k items using structured prompts.
  arXiv  Detail & Related papers  (2025-05-08T05:01:44Z)
• Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping [2.427844597259453]
  Reinforcement learning (RL) often struggles with reward misalignment.<n>Human-in-the-loop (HITL) methods can mitigate this issue, but they also introduce biases.<n>We propose two key contributions to address these challenges.
  arXiv  Detail & Related papers  (2025-03-26T03:17:12Z)
• WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback [36.06000681394939]
  We introduce WildFeedback, a novel framework that leverages in-situ user feedback during conversations with large language models (LLMs) to create preference datasets automatically.<n>Our experiments demonstrate that LLMs fine-tuned on WildFeedback dataset exhibit significantly improved alignment with user preferences.
  arXiv  Detail & Related papers  (2024-08-28T05:53:46Z)
• Prompt Optimization with Human Feedback [69.95991134172282]
  We study the problem of prompt optimization with human feedback (POHF) We introduce our algorithm named automated POHF (APOHF) The results demonstrate that our APOHF can efficiently find a good prompt using a small number of preference feedback instances.
  arXiv  Detail & Related papers  (2024-05-27T16:49:29Z)
• Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models [61.45529177682614]
  We challenge the prevailing constrained evaluation paradigm for values and opinions in large language models. We show that models give substantively different answers when not forced. We distill these findings into recommendations and open challenges in evaluating values and opinions in LLMs.
  arXiv  Detail & Related papers  (2024-02-26T18:00:49Z)
• LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback [65.84061725174269]
  Recent large language models (LLM) are leveraging human feedback to improve their generation quality. We propose LLMRefine, an inference time optimization method to refine LLM's output. We conduct experiments on three text generation tasks, including machine translation, long-form question answering (QA), and topical summarization. LLMRefine consistently outperforms all baseline approaches, achieving improvements up to 1.7 MetricX points on translation tasks, 8.1 ROUGE-L on ASQA, 2.2 ROUGE-L on topical summarization.
  arXiv  Detail & Related papers  (2023-11-15T19:52:11Z)
• UltraFeedback: Boosting Language Models with Scaled AI Feedback [99.4633351133207]
  We present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models.
  arXiv  Detail & Related papers  (2023-10-02T17:40:01Z)
• Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models [32.843361525236965]
  We analyze the effect of sparse feedback on the alignment and evaluation of large language models. We find that preferences from ratings and rankings significantly disagree 60% for both human and AI annotators. Our findings shed light on critical gaps in methods for evaluating the real-world utility of language models.
  arXiv  Detail & Related papers  (2023-08-30T07:35:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.