Related papers: ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

URL: http://arxiv.org/abs/2605.20087v2
Date: Thu, 21 May 2026 22:24:42 GMT
Title: ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions
Abstract summary: ThoughtTrace is the first large-scale dataset that pairs real-world multi-turn human--AI conversations with users' self-reported thoughts.<n>ThoughtTrace comprises 1,058 users, 2,155 conversations, 17,058 turns, and 10,174 thought annotations collected across 20 language models.
Score: 31.70995208851302
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Conversational AI has now reached billions of users, yet existing datasets capture only what people say, not what they think. We introduce ThoughtTrace, the first large-scale dataset that pairs real-world multi-turn human--AI conversations with users' self-reported thoughts: their reasons for sending prompts and reactions to assistant responses. ThoughtTrace comprises 1,058 users, 2,155 conversations, 17,058 turns, and 10,174 thought annotations collected across 20 language models. Our analysis shows that ThoughtTrace captures long-horizon, topically diverse interactions, and that thoughts are semantically distinct from messages, difficult for frontier LLMs to infer from context, diverse in content, and tied to conversation stages. We further demonstrate the utility of thoughts for downstream modeling. First, thoughts improve user-behavior prediction as inference-time context. Second, thought-guided rewrites provide fine-grained alignment signals for training personalized assistants. Together, ThoughtTrace establishes user thoughts as a new data modality for studying the cognitive dynamics behind human--AI interaction and provides a foundation for building assistants that better understand and adapt to users' latent goals, preferences, and needs.

Related papers

Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States [72.77158276658636]
Mind2Dialogue is a framework for simulating users' mental states and turning them into privileged supervision for human-aware training.<n>We first propose a psychology-guided simulator that preserves personal characteristics while updating mental states to generate coherent conversations.<n>Our privileged distillation then trains models on the Oracle's well-informed responses to assist users without direct access to their mental states at deployment.
arXiv Detail & Related papers (2026-09-14T17:55:58Z)
Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation [60.96010213186819]
We introduce Interactive Tracking, a new paradigm that allows users to guide the tracker at any time using natural language commands.<n>We present InteractTrack, the first large-scale benchmark for interactive tracking, containing 150 videos with dense bounding box annotations and timestamped language instructions.<n>Third, we introduce Interactive Memory-Augmented Tracking (IMAT), a new baseline that employs a dynamic memory mechanism to learn from user feedback and update tracking behavior accordingly.
arXiv Detail & Related papers (2026-04-02T12:33:27Z)
FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback [92.67587639164908]
We present FronTalk, a benchmark for front-end code generation with multi-modal feedback.<n>We focus on the front-end development task and curate FronTalk, a collection of 100 multi-turn dialogues.<n> Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature.
arXiv Detail & Related papers (2025-12-05T23:28:09Z)
Human vs. Agent in Task-Oriented Conversations [22.743152820695588]
This work presents the first systematic comparison between large language models (LLMs)-simulated users and human users in personalized task-oriented conversations.<n>Our analysis reveals significant behavioral differences between the two user types in problem-solving approaches.
arXiv Detail & Related papers (2025-09-22T11:30:39Z)
Thinking Before You Speak: A Proactive Test-time Scaling Approach [54.8205006555199]
We implement our idea as a reasoning framework, named emphThinking Before You Speak (TBYS)<n>We design a pipeline for automatically collecting and filtering in-context examples for the generation of emphinsights.<n>Experiments on challenging mathematical datasets verify the effectiveness of TBYS.
arXiv Detail & Related papers (2025-08-26T03:43:32Z)
REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation [51.97224538045096]
We introduce REALTALK, a 21-day corpus of authentic messaging app dialogues.<n>We compare EI attributes and persona consistency to understand the challenges posed by real-world dialogues.<n>Our findings reveal that models struggle to simulate a user solely from dialogue history, while fine-tuning on specific user chats improves persona emulation.
arXiv Detail & Related papers (2025-02-18T20:29:01Z)
Proactive Conversational Agents with Inner Thoughts [42.34140072445951]
One of the long-standing aspirations in conversational AI is to allow them to autonomously take initiatives in conversations, i.e., being proactive.<n>Prior NLP research focused mainly on predicting the next speaker from contexts like preceding conversations.<n>We propose that just like humans, rather than merely reacting to turn-taking cues, a proactive AI formulates its own inner thoughts during a conversation, and seeks the right moment to contribute.
arXiv Detail & Related papers (2024-12-31T10:41:56Z)
UltraFeedback: Boosting Language Models with Scaled AI Feedback [99.4633351133207]
We present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models.
arXiv Detail & Related papers (2023-10-02T17:40:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.