From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP

• URL: http://arxiv.org/abs/2601.07988v1
• Date: Mon, 12 Jan 2026 20:38:36 GMT
• Title: From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP
• Abstract summary: We show that document-level evaluation can yield substantially different and sometimes reversed conclusions compared to our ecologically valid modeling and evaluation.<n>We tie our results to a broader discussion motivating a shift from word-sequence evaluation toward $textitbehavior-sequence$ paradigms for NLP.
• Score: 10.614902538213638
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While NLP typically treats documents as independent and unordered samples, in longitudinal studies, this assumption rarely holds: documents are nested within authors and ordered in time, forming person-indexed, time-ordered $\textit{behavioral sequences}$. Here, we demonstrate the need for and propose a longitudinal modeling and evaluation paradigm that consequently updates four parts of the NLP pipeline: (1) evaluation splits aligned to generalization over people ($\textit{cross-sectional}$) and/or time ($\textit{prospective}$); (2) accuracy metrics separating between-person differences from within-person dynamics; (3) sequence inputs to incorporate history by default; and (4) model internals that support different $\textit{coarseness}$ of latent state over histories (pooled summaries, explicit dynamics, or interaction-based models). We demonstrate the issues ensued by traditional pipeline and our proposed improvements on a dataset of 17k daily diary transcripts paired with PTSD symptom severity from 238 participants, finding that traditional document-level evaluation can yield substantially different and sometimes reversed conclusions compared to our ecologically valid modeling and evaluation. We tie our results to a broader discussion motivating a shift from word-sequence evaluation toward $\textit{behavior-sequence}$ paradigms for NLP.

Related papers

• RecRec: Latent Interests Recursive Reasoning for Sequential Recommendation [52.23179023639867]
  Sequential recommender systems rely on a single forward pass to encode user interaction and predict the next item.<n>How to structure the reasoning process for sequential recommendation remains an open question.<n>Existing approaches couple reasoning and prediction in a single $d$-dimensional state, limiting reasoning depth.<n>We propose RecRec (Recursive Reasoning for Recommendation), an RL-free framework that overcomes fixed $d$-dimensional state of methods.
  arXiv  Detail & Related papers  (2026-07-14T16:28:28Z)
• Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics [49.443264461057645]
  Diffusion and continuous flow-based language models have emerged as the leading non-autoregressive alternatives to language modeling.<n>Progress in both paradigms is overwhelmingly tracked by generative perplexity (gen-PPL): the per-token negative log-likelihood of samples under a frozen autoregressive (AR) scorer.<n>We argue that this metric is unsound. By construction, gen-PPL measures only predictability under the scoring AR, not grammaticality or semantic coherence.
  arXiv  Detail & Related papers  (2026-06-07T02:35:56Z)
• BehaviorBench: Modeling Real-World User Decisions from Behavioral Traces [53.55407626944686]
  textscBehaviorBench is a benchmark for evaluating personalized decision modeling from real-world behavioral traces.<n>textscBehaviorBench reconstructs wallet-level decision histories from observed public prediction-market and on-chain records.<n>Across 2,000 evaluation wallets, the benchmark contains 141,445 Belief instances and 1,485,972 Trade instances.
  arXiv  Detail & Related papers  (2026-06-01T19:04:36Z)
• SCORE: A Semantic Evaluation Framework for Generative Document Parsing [2.5101597298392098]
  Multi-modal generative document parsing systems produce semantically correct yet structurally divergent outputs.<n>Conventional metrics-CER, WER, IoU, or TEDS-misclassify such diversity as error, penalizing valid interpretations and obscuring system behavior.<n>We introduce SCORE, an interpretation-agnostic framework that integrates (i) adjusted edit distance for robust content fidelity, (ii) token-level diagnostics to distinguish hallucinations from omissions, (iii) table evaluation with spatial tolerance and semantic alignment, and (iv) hierarchy-aware consistency checks.
  arXiv  Detail & Related papers  (2025-09-16T16:06:19Z)
• CoKe: Customizable Fine-Grained Story Evaluation via Chain-of-Keyword Rationalization [90.15027447565427]
  Chain of thought (CoT) generates free-text explanations that help guide a model's predictions.<n>Self-Consistency (SC) marginalizes predictions over multiple generated explanations.<n>We propose $textbfC$hain-$textbfo$f-$textbfKe$ywords (CoKe)
  arXiv  Detail & Related papers  (2025-03-21T13:37:46Z)
• How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators [8.244694683982784]
  We investigate the questions of assessing the performance of human annotators and incentivizing them to provide high-quality annotations.<n>We develop a principal-agent model to characterize the behaviors of and the interactions between the company and the human annotators.<n>The model rationalizes a practical mechanism of a bonus scheme to incentivize annotators which benefits both parties.
  arXiv  Detail & Related papers  (2025-02-10T12:15:27Z)
• Localizing Factual Inconsistencies in Attributable Text Generation [74.11403803488643]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation.<n>We show that QASemConsistency yields factual consistency scores that correlate well with human judgments.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• BBScoreV2: Learning Time-Evolution and Latent Alignment from Stochastic Representation [23.765789561546715]
  Autoregressive generative models play a key role in various language tasks, especially for modeling and evaluating long text sequences.<n>In this work, we observe that fitting transformer-based model embeddings into a process yields ordered latent representations from originally unordered model outputs.<n>We introduce a novel likelihood-based evaluation metric BBVScore2, offering both intuitive and quantitative support for the effectiveness of BBV2.
  arXiv  Detail & Related papers  (2024-05-28T02:33:38Z)
• Higher-Order DeepTrails: Unified Approach to *Trails [7.270112855088838]
  Analyzing, understanding, and describing human behavior is advantageous in different settings, such as web browsing or traffic navigation. We propose to analyze entire sequences using autoregressive language models, as they are traditionally used to model higher-order dependencies in sequences. We show that our approach can be easily adapted to model different settings introduced in previous work, namely HypTrails, MixedTrails and even SubTrails.
  arXiv  Detail & Related papers  (2023-10-06T06:54:11Z)
• DSI++: Updating Transformer Memory with New Documents [95.70264288158766]
  We introduce DSI++, a continual learning challenge for DSI to incrementally index new documents. We show that continual indexing of new documents leads to considerable forgetting of previously indexed documents. We introduce a generative memory to sample pseudo-queries for documents and supplement them during continual indexing to prevent forgetting for the retrieval task.
  arXiv  Detail & Related papers  (2022-12-19T18:59:34Z)
• Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality [84.94877848357896]
  Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models. We analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual exclusivity bias and the tendency to memorize whole examples. We show substantial empirical improvements using standard sequence-to-sequence models on two widely-used compositionality datasets.
  arXiv  Detail & Related papers  (2022-11-28T17:36:41Z)
• Document-Level Relation Extraction with Sentences Importance Estimation and Focusing [52.069206266557266]
  Document-level relation extraction (DocRE) aims to determine the relation between two entities from a document of multiple sentences. We propose a Sentence Estimation and Focusing (SIEF) framework for DocRE, where we design a sentence importance score and a sentence focusing loss. Experimental results on two domains show that our SIEF not only improves overall performance, but also makes DocRE models more robust.
  arXiv  Detail & Related papers  (2022-04-27T03:20:07Z)
• Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters [35.103851212995046]
  Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs. We explore the direct zero-shot applicability of NLI models to real applications, beyond the sentence-pair setting they were trained on. We develop new aggregation methods to allow operating over full documents, reaching state-of-the-art performance on the ContractNLI dataset.
  arXiv  Detail & Related papers  (2022-04-15T12:56:39Z)
• An Empirical Study: Extensive Deep Temporal Point Process [61.14164208094238]
  We first review recent research emphasis and difficulties in modeling asynchronous event sequences with deep temporal point process.<n>We propose a Granger causality discovery framework for exploiting the relations among multi-types of events.
  arXiv  Detail & Related papers  (2021-10-19T10:15:00Z)
• Sequence Adaptation via Reinforcement Learning in Recommender Systems [8.909115457491522]
  We propose the SAR model, which learns the sequential patterns and adjusts the sequence length of user-item interactions in a personalized manner. In addition, we optimize a joint loss function to align the accuracy of the sequential recommendations with the expected cumulative rewards of the critic network. Our experimental evaluation on four real-world datasets demonstrates the superiority of our proposed model over several baseline approaches.
  arXiv  Detail & Related papers  (2021-07-31T13:56:46Z)
• Author Clustering and Topic Estimation for Short Texts [69.54017251622211]
  We propose a novel model that expands on the Latent Dirichlet Allocation by modeling strong dependence among the words in the same document. We also simultaneously cluster users, removing the need for post-hoc cluster estimation. Our method performs as well as -- or better -- than traditional approaches to problems arising in short text.
  arXiv  Detail & Related papers  (2021-06-15T20:55:55Z)
• Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization [72.54873655114844]
  Text summarization is one of the most challenging and interesting problems in NLP. This work proposes a multi-view sequence-to-sequence model by first extracting conversational structures of unstructured daily chats from different views to represent conversations. Experiments on a large-scale dialogue summarization corpus demonstrated that our methods significantly outperformed previous state-of-the-art models via both automatic evaluations and human judgment.
  arXiv  Detail & Related papers  (2020-10-04T20:12:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.