Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarks

• URL: http://arxiv.org/abs/2603.04532v1
• Date: Wed, 04 Mar 2026 19:18:11 GMT
• Title: Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarks
• Abstract summary: temporal changes in technical corpora, such as API deprecations and code reorganizations, can render existing benchmarks stale.<n>We examine two independent corpus snapshots of FreshStack from October 2024 and October 2025 to answer questions about LangChain.
• Score: 40.92390378341581
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Information retrieval (IR) benchmarks typically follow the Cranfield paradigm, relying on static and predefined corpora. However, temporal changes in technical corpora, such as API deprecations and code reorganizations, can render existing benchmarks stale. In our work, we investigate how temporal corpus drift affects FreshStack, a retrieval benchmark focused on technical domains. We examine two independent corpus snapshots of FreshStack from October 2024 and October 2025 to answer questions about LangChain. Our analysis shows that all but one query posed in 2024 remain fully supported by the 2025 corpus, as relevant documents "migrate" from LangChain to competitor repositories, such as LlamaIndex. Next, we compare the accuracy of retrieval models on both snapshots and observe only minor shifts in model rankings, with overall strong correlation of up to 0.978 Kendall $τ$ at Recall@50. These results suggest that retrieval benchmarks re-judged with evolving temporal corpora can remain reliable for retrieval evaluation. We publicly release all our artifacts at https://github.com/fresh-stack/driftbench.

Related papers

• VisDocAgentBench: Benchmarking Agents for Visually Rich Document Retrieval [65.89247522243959]
  VisDocAgentBench is a closed-corpus benchmark comparing static and agentic retrieval under a ranked-output contract.<n>It contains 2,375 pages from 100 documents and 120 unique-target queries balanced across direct, one-bridge, and two-bridge evidence structures.<n>A strong late-interaction visual retriever reaches 97.50% Recall@1 on direct items but 2.50% on two-bridge items, exposing the limits of query--target matching when relevance depends on corpus context.
  arXiv  Detail & Related papers  (2026-08-18T15:23:06Z)
• Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law [0.0]
  We identify and quantify temporal misgrounding: the systematic retrieval and citation of the currently in-force version of a legal article.<n>We introduce FiscalQA Pro, pairing a versioned corpus of 32,436 article-versions of the French tax code with an all-model-hard temporal-reasoning track.<n>Our end-to-end retriever over a multi-version index, with no oracle, reaches 98.3% mean strict.
  arXiv  Detail & Related papers  (2026-08-10T10:20:13Z)
• Submitted and Diagnostic Analysis of Full-Text Temporal Retrieval for LongEval-Sci [1.0312968200748118]
  LongEval-Sci evaluates scientific retrieval under collection change.<n>This paper reports both official Task 1 results and development diagnostics for LongEval-Sci 2026.
  arXiv  Detail & Related papers  (2026-07-05T02:52:30Z)
• Temporal Preference Optimization for Unsupervised Retrieval [11.583736130556611]
  We propose TPOUR (Temporal Preference Optimization for Unsupervised Retriever), which uses our novel training method Temporal Retrieval Preference Optimization (TRPO)<n>TPOUR generalizes to unseen time periods via in a learned time embedding, enabling continuous temporal alignment.<n>Compared to Qwen-Embedding-8B, despite being about 72.7x smaller, TPOUR Contriever improves average nDCG@5 by +4.04 (+12.15%) on explicit and +4.98 (+15.21%) on implicit queries.
  arXiv  Detail & Related papers  (2026-06-16T08:27:22Z)
• An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendation [50.09718257952108]
  Sequentialsolv has increasingly shifted toward generative recommenders that combine sequential patterns with semantic item information.<n>These methods are often evaluated on a small set of widely used benchmarks, raising a key question: do these benchmarks actually require the advanced modeling capabilities that modern generative recommenders claim to provide?<n>We conduct a benchmark audit with an intentionally simple graph. Starting from only the last one or two items, it retrieves candidates from a few-hop item-transition graph and ranks them by item-feature similarity.
  arXiv  Detail & Related papers  (2026-05-08T02:00:11Z)
• FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation [73.22935457705057]
  We introduce FRESCO, a benchmark for evaluating re-rankers in temporally dynamic contexts.<n>By pairing recency-seeking queries with historical Wikipedia revisions, FRESCO tests whether re-rankers can prioritize factually recent evidence while maintaining semantic relevance.<n>Our evaluation reveals a consistent failure mode across existing re-rankers: a strong bias toward older, semantically rich documents, even when they are factually obsolete.
  arXiv  Detail & Related papers  (2026-04-14T17:04:25Z)
• Benchmarking Real-Time Question Answering via Executable Code Workflows [15.866566641224715]
  RT-QA is a dynamic evaluation framework that leverages executable code to retrieve up-to-date answers at evaluation time.<n>We construct an agent-driven pipeline that autonomously generates code for web crawling and DOM-based answer extraction.<n> RT-QA spans 12 domains with 320 Chinese questions categorized into three difficulty levels.
  arXiv  Detail & Related papers  (2026-03-16T16:20:29Z)
• RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems [6.083097040417168]
  We introduce LR-bench, a benchmark curated from 2024-2025 AI/NLP manuscripts with five-level self-assessed familiarity ratings.<n>We also propose RATE, a reviewer-centric ranking framework that distills each reviewer's recent publications into compact keyword-based profiles.<n>Our approach consistently achieves state-of-the-art performance, outperforming strong embedding baselines by a clear margin.
  arXiv  Detail & Related papers  (2026-01-27T14:13:46Z)
• TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrieval [44.94371780739013]
  Existing temporal QA benchmarks focus on fact-seeking queries from news corpora.<n>We introduce TEMPO, the first benchmark combining temporal reasoning with reasoning-intensive retrieval across 13 domains.
  arXiv  Detail & Related papers  (2026-01-14T14:45:20Z)
• LiveVectorLake: A Real-Time Versioned Knowledge Base Architecture for Streaming Vector Updates and Temporal Retrieval [0.0]
  LivevusLake is a dual-tier temporal knowledge base architecture that enables real-time semantic search on current knowledge.<n>System maintains complete version history for compliance, auditability, and point-in-time retrieval.
  arXiv  Detail & Related papers  (2025-11-24T11:15:39Z)
• LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge [31.40589987269264]
  We present LiveSearchBench, an automated pipeline for constructing retrieval-dependent benchmarks from recent knowledge updates.<n>Our method computes deltas between successive Wikidata snapshots, filters candidate triples for quality, and synthesizes natural-language questions at three levels of reasoning difficulty.<n> Experiments show a pronounced performance drop when models confront facts that post-date pretraining, with the gap most salient on multi-hop queries.
  arXiv  Detail & Related papers  (2025-11-03T10:00:49Z)
• DS@GT at LongEval: Evaluating Temporal Performance in Web Search Systems and Topics with Two-Stage Retrieval [44.99833362998488]
  The DS@GT competition team participated in the Longitudinal Evaluation of Model Performance (LongEval) lab at CLEF 2025.<n>Our analysis of the Qwant web dataset includes exploratory data analysis with topic modeling over time.<n>Our best system achieves an average NDCG@10 of 0.296 across the entire training and test dataset, with an overall best score of 0.395 on 2023-05.
  arXiv  Detail & Related papers  (2025-07-11T07:23:08Z)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents.<n>Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.<n>We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)
• Towards Real-World Visual Tracking with Temporal Contexts [64.7981374129495]
  We propose a two-level framework (TCTrack) that can exploit temporal contexts efficiently. Based on it, we propose a stronger version for real-world visual tracking, i.e., TCTrack++. For feature extraction, we propose an attention-based temporally adaptive convolution to enhance the spatial features. For similarity map refinement, we introduce an adaptive temporal transformer to encode the temporal knowledge efficiently.
  arXiv  Detail & Related papers  (2023-08-20T17:59:40Z)
• Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation [89.88214896713846]
  STA score considers two critical factors: temporal redundancy and semantic importance. We apply the STA module to off-the-shelf video Transformers and Videowins. Results: Kinetics-400 and Something-Something V2 achieve 30% overshelf reduction with a negligible 0.2% accuracy drop.
  arXiv  Detail & Related papers  (2023-08-08T19:38:15Z)
• TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement [64.11385310305612]
  We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages: (1) a matching stage, which independently locates a suitable candidate point match for the query point on every other frame, and (2) a refinement stage, which updates both the trajectory and query features based on local correlations. The resulting model surpasses all baseline methods by a significant margin on the TAP-Vid benchmark, as demonstrated by an approximate 20% absolute average Jaccard (AJ) improvement on DAVIS.
  arXiv  Detail & Related papers  (2023-06-14T17:07:51Z)
• ReFIT: Relevance Feedback from a Reranker during Inference [109.33278799999582]
  Retrieve-and-rerank is a prevalent framework in neural information retrieval. We propose to leverage the reranker to improve recall by making it provide relevance feedback to the retriever at inference time.
  arXiv  Detail & Related papers  (2023-05-19T15:30:33Z)
• Understanding Image Retrieval Re-Ranking: A Graph Neural Network Perspective [52.96911968968888]
  In this paper, we demonstrate that re-ranking can be reformulated as a high-parallelism Graph Neural Network (GNN) function. On the Market-1501 dataset, we accelerate the re-ranking processing from 89.2s to 9.4ms with one K40m GPU, facilitating the real-time post-processing.
  arXiv  Detail & Related papers  (2020-12-14T15:12:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.