Reproduction Beyond Benchmarks: ConstBERT and ColBERT-v2 Across Backends and Query Distributions

• URL: http://arxiv.org/abs/2604.09982v2
• Date: Thu, 16 Apr 2026 04:09:13 GMT
• Title: Reproduction Beyond Benchmarks: ConstBERT and ColBERT-v2 Across Backends and Query Distributions
• Abstract summary: We evaluate ColBERT-v2 and ConstBERT across five dimensions.<n>While ConstBERT reproduces within 0.05% MRR@10 on MS-MARCO, both models show a drop of 86-97% on long, narrative queries.
• Score: 3.976291254896486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reproducibility must validate architectural robustness, not just numerical accuracy. We evaluate ColBERT-v2 and ConstBERT across five dimensions, finding that while ConstBERT reproduces within 0.05% MRR@10 on MS-MARCO, both models show a drop of 86-97% on long, narrative queries (TREC ToT 2025). Ablations prove this failure is architectural: performance plateaus at 20 words because the MaxSim operator's uniform token weighting cannot distinguish signal from filler noise. Furthermore, undocumented backend parameters create an 8-point gap due to ConstBERT's sparse centroid coverage, and fine-tuning with 3x more data actually degrades performance by up to 29%. We conclude that architectural constraints in multi-vector retrieval cannot be overcome by adaptation alone. Code: https://github.com/utshabkg/multi-vector-reproducibility.

Related papers

• Pre-Whitening and BCJR Posterior Distillation for Bi-LSTM Detection in Faster-than-Nyquist Signaling [0.0]
  Recurrent detectors are low-complexity alternatives to the optimal Bahl-Cocke-Jelinek-Raviv (BCJR) detector for faster-than-Nyquist (FTN) signaling.<n>We show that processing nested ISI windows in separate recurrent branches improves the bit error rate (BER) of a Bi-LSTM network.
  arXiv  Detail & Related papers  (2026-09-07T17:07:49Z)
• MoganColBERT-TR: A Late-Interaction Multi-Vector Retrieval Model for Turkish [0.0]
  This work introduces the third model in that lineage: MoganColBERT-TR.<n>MoganColBERT-TR is a multi-vector retrieval model that represents it at the token level through a 768->128 projection and scores it with MaxSim late interaction.<n>With 148.9M parameters, MoganColBERT-TR reaches an overall score of 37.36.
  arXiv  Detail & Related papers  (2026-08-26T19:26:15Z)
• Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection [0.33842793760651557]
  Existing research shows that AI-generated text detection classifiers achieve strong in-distribution (ID) performance but do not maintain the same performance on out-of-distribution (OOD) texts.<n>We fine-tune BERT-tiny models with Bayesian classification heads to select texts across three different datasets to use as a consolidated training set.<n>Our results highlight that careful dataset curation can lead to strong OOD performance.
  arXiv  Detail & Related papers  (2026-07-19T18:48:59Z)
• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• vstash: Local-First Hybrid Retrieval with Adaptive Fusion for LLM Agents [0.0]
  We present **vstash**, a local-first document memory system that combines vector similarity search with fulltext keyword matching.<n>All data resides in a single file using sqlite-vec for approximate nearest neighbor search and FTS5 for keyword matching.
  arXiv  Detail & Related papers  (2026-04-16T19:22:58Z)
• TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish [0.7233065479782755]
  TabiBERT is a monolingual Turkish encoder based on ModernBERT architecture trained from scratch on a large, curated corpus.<n>It supports 8,192-token context length (16x original BERT), achieves up to 2.65x speedup, and reduces GPU memory consumption.<n>It attains 77.58 on TabiBench, outperforming BERTurk by 1.62 points and establishing state-of-the-art on five of eight categories.
  arXiv  Detail & Related papers  (2025-12-28T20:18:22Z)
• Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction [95.91743732150233]
  Goedel-Prover-V2, a series of open-source language models, set a new state-of-the-art in automated theorem proving.<n>We generate synthetic tasks of increasing difficulty to train the model to master increasingly complex theorems.<n>Goedel-Prover-V2-32B achieves 88.1% on MiniF2F at pass@32 in standard mode and 90.4% in self-correction mode.
  arXiv  Detail & Related papers  (2025-08-05T16:28:22Z)
• Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever [6.221757399678299]
  ColBERT's late interaction scoring approximates the joint query-document attention seen in cross-encoders. Our new model, Jina-ColBERT-v2, demonstrates strong performance across a range of English and multilingual retrieval tasks.
  arXiv  Detail & Related papers  (2024-08-29T16:21:00Z)
• Shallow Cross-Encoders for Low-Latency Retrieval [69.06104373460597]
  Cross-Encoders based on large transformer models (such as BERT or T5) are computationally expensive and allow for scoring only a small number of documents within a reasonably small latency window. We show that weaker shallow transformer models (i.e., transformers with a limited number of layers) actually perform better than full-scale models when constrained to these practical low-latency settings.
  arXiv  Detail & Related papers  (2024-03-29T15:07:21Z)
• Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT [48.35407228760352]
  Retrieval pipelines perform poorly in domains where documents are long (e.g., 10K tokens or more) and where identifying the relevant document requires synthesizing information across the entire text. We develop long-context retrieval encoders suitable for these domains. We first introduce LoCoV1, a novel 12 task benchmark constructed to measure long-context retrieval where chunking is not possible or not effective. We next present the M2-BERT retrieval encoder, an 80M parameter state-space encoder model built from the Monarch Mixer architecture, capable of scaling to documents up to 32K tokens long.
  arXiv  Detail & Related papers  (2024-02-12T06:43:52Z)
• EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation [82.3956677850676]
  Pre-trained language models have shown remarkable results on various NLP tasks. Due to their bulky size and slow inference speed, it is hard to deploy them on edge devices. In this paper, we have a critical insight that improving the feed-forward network (FFN) in BERT has a higher gain than improving the multi-head attention (MHA)
  arXiv  Detail & Related papers  (2021-09-15T11:25:39Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.