Precise Legal Sentence Boundary Detection for Retrieval at Scale: NUPunkt and CharBoundary

• URL: http://arxiv.org/abs/2504.04131v1
• Date: Sat, 05 Apr 2025 10:48:34 GMT
• Title: Precise Legal Sentence Boundary Detection for Retrieval at Scale: NUPunkt and CharBoundary
• Abstract summary: We present NUPunkt and CharBoundary, two sentence boundary detection libraries optimized for high-precision processing of legal text.<n>These libraries address the critical challenges posed by legal documents containing specialized citations, abbreviations, and complex sentence structures.
• Score: 13.284214798785847
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present NUPunkt and CharBoundary, two sentence boundary detection libraries optimized for high-precision, high-throughput processing of legal text in large-scale applications such as due diligence, e-discovery, and legal research. These libraries address the critical challenges posed by legal documents containing specialized citations, abbreviations, and complex sentence structures that confound general-purpose sentence boundary detectors. Our experimental evaluation on five diverse legal datasets comprising over 25,000 documents and 197,000 annotated sentence boundaries demonstrates that NUPunkt achieves 91.1% precision while processing 10 million characters per second with modest memory requirements (432 MB). CharBoundary models offer balanced and adjustable precision-recall tradeoffs, with the large model achieving the highest F1 score (0.782) among all tested methods. Notably, NUPunkt provides a 29-32% precision improvement over general-purpose tools while maintaining exceptional throughput, processing multi-million document collections in minutes rather than hours. Both libraries run efficiently on standard CPU hardware without requiring specialized accelerators. NUPunkt is implemented in pure Python with zero external dependencies, while CharBoundary relies only on scikit-learn and optional ONNX runtime integration for optimized performance. Both libraries are available under the MIT license, can be installed via PyPI, and can be interactively tested at https://sentences.aleainstitute.ai/. These libraries address critical precision issues in retrieval-augmented generation systems by preserving coherent legal concepts across sentences, where each percentage improvement in precision yields exponentially greater reductions in context fragmentation, creating cascading benefits throughout retrieval pipelines and significantly enhancing downstream reasoning quality.

Related papers

• REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling [0.0]
  REIGN is a bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network (GN) rather than on raw tokens.<n>We release a synthetic long-document retrieval benchmark for contrastive training and evaluation at long context lengths.<n>A paired significance test puts it on par with models 1.6-4.3x larger on the patent task, and it stays within 0.65 nDCG@10 of a 20x-larger model on LoCo.
  arXiv  Detail & Related papers  (2026-08-30T16:50:38Z)
• Real-Time Hybrid Retrieval in Hyperbolic Space for Retrieval-Augmented Generation on Edge Devices [0.0]
  This paper presents a hybrid document retrieval system that operates entirely within the Lorentz model of hyperbolic geometry.<n>Unlike conventional dense retrievers confined to Euclidean space, this system projects pretrained word embeddings into hyperbolic space.<n>The system supports real-time indexing of user-supplied documents and resource-efficient querying over tens of thousands of moderately sized documents.
  arXiv  Detail & Related papers  (2026-08-02T19:14:59Z)
• Efficient and Scalable Provenance Tracking for LLM-Generated Code Snippets [3.5312864406384485]
  Large language models (LLMs) for code completion and generation may reproduce training examples verbatim and without authorship attribution.<n> Classical fingerprint-based plagiarism detectors based on fingerprinting, such as Winnowing, remain highly effective.<n>We introduce a 300M- parameter encoder tailored for code retrieval, together with a hybrid two-stage provenance-tracking pipeline.
  arXiv  Detail & Related papers  (2026-05-27T14:12:17Z)
• Towards Intelligent Legal Document Analysis: CNN-Driven Classification of Case Law Texts [1.3192560874022086]
  This work presents a lightweight yet high-accuracy framework for citation-treatment classification.<n>It pairs lemmatisation-based preprocessing with subword-aware FastText embeddings and a one-dimensional Convolutional Neural Network (CNN)<n>The proposed system achieves 97.26% classification accuracy and a macro F1-score of 96.82%, surpassing established baselines.
  arXiv  Detail & Related papers  (2026-04-20T00:14:11Z)
• Residual Context Diffusion Language Models [90.07635240595926]
  Residual Context Diffusion (RCD) is a module that converts discarded token representations into contextual residuals and injects them back for the next denoising step.<n>RCD consistently improves frontier dLLMs by 5-10 points in accuracy with minimal extra computation overhead.
  arXiv  Detail & Related papers  (2026-01-30T13:16:32Z)
• Migrating QAOA from Qiskit 1.x to 2.x: An experience report [0.3679863727963206]
  Migrating quantum algorithms across evolving frameworks introduces subtle behavioral changes that affect accuracy and algorithms.<n>This paper reports our experience converting the Quantum Algorithm Optimization (QAOA) from Qiskit with Qiskit 1.x (v1 primitives) to a custom implementation using Qiskit 2.x (v2 primitives)<n>Despite identical circuits, Approximates, and Hamiltonians, the new version produced drastically different results.
  arXiv  Detail & Related papers  (2025-12-09T04:58:56Z)
• APEX: Approximate-but-exhaustive search for ultra-large combinatorial synthesis libraries [0.28106259549258145]
  Make-on-demand synthesis libraries (CSLs) like Enamine REAL have significantly enabled drug discovery efforts.<n>Currently, a routine virtual screening campaign may be limited to scoring fewer than 0.1% of the available compounds.<n>We propose the approximate-but-exhaustive search protocol for CSLs, or APEX.
  arXiv  Detail & Related papers  (2025-10-28T12:57:59Z)
• Transform Before You Query: A Privacy-Preserving Approach for Vector Retrieval with Embedding Space Alignment [7.491164990682839]
  STEER (textbfSecure textbfTransformed textbfEmbedding vtextbfEctortextbf Retrieval) is a private vector retrieval framework.<n>It safeguards query text privacy while maintaining the retrieval accuracy.
  arXiv  Detail & Related papers  (2025-07-24T15:41:34Z)
• Design of intelligent proofreading system for English translation based on CNN and BERT [5.498056383808144]
  This paper proposes a novel hybrid approach for robust proofreading.<n>It combines convolutional neural networks (CNN) with Bidirectional Representations from Transformers (BERT)<n> Experiments attain a 90% accuracy, 89.37% F1, and 16.24% MSE, exceeding recent proofreading techniques by over 10% overall.
  arXiv  Detail & Related papers  (2025-06-05T09:34:42Z)
• Case Study: Fine-tuning Small Language Models for Accurate and Private CWE Detection in Python Code [0.0]
  Large Language Models (LLMs) have demonstrated significant capabilities in understanding and analyzing code for security vulnerabilities. This work explores the potential of Small Language Models (SLMs) as a viable alternative for accurate, on-premise vulnerability detection.
  arXiv  Detail & Related papers  (2025-04-23T10:05:27Z)
• DLaVA: Document Language and Vision Assistant for Answer Localization with Enhanced Interpretability and Trustworthiness [34.170341753045776]
  Document Visual Question Answering (VQA) demands robust integration of text detection, recognition, and spatial reasoning.<n>We introduce DLaVA, a training-free pipeline that leverages Multimodal Large Language Models (MLLMs) for zero-shot answer localization.
  arXiv  Detail & Related papers  (2024-11-29T06:17:11Z)
• Detecting Multi-Parameter Constraint Inconsistencies in Python Data Science Libraries [21.662640566736098]
  We propose MPDetector, for detecting inconsistencies between code and documentation. MPDetector identifies these constraints at the code level by exploring execution paths through symbolic execution. We propose a customized fuzzy constraint logic to reconcile the unpredictability of LLM outputs.
  arXiv  Detail & Related papers  (2024-11-18T09:30:14Z)
• Linear-time Minimum Bayes Risk Decoding with Reference Aggregation [52.1701152610258]
  Minimum Bayes Risk (MBR) decoding is a text generation technique that has been shown to improve the quality of machine translations. It requires the pairwise calculation of a utility metric, which has quadratic complexity. We propose to approximate pairwise metric scores with scores calculated against aggregated reference representations.
  arXiv  Detail & Related papers  (2024-02-06T18:59:30Z)
• Advancing Hungarian Text Processing with HuSpaCy: Efficient and Accurate NLP Pipelines [0.0]
  This paper presents a set of industrial-grade text processing models for Hungarian. Models have been implemented in the spaCy framework, extending the HuSpaCy toolkit. All experiments are reproducible and the pipelines are freely available under a permissive license.
  arXiv  Detail & Related papers  (2023-08-24T08:19:51Z)
• SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval [92.27387459751309]
  We provide SPRINT, a unified Python toolkit for evaluating neural sparse retrieval. We establish strong and reproducible zero-shot sparse retrieval baselines across the well-acknowledged benchmark, BEIR. We show that SPLADEv2 produces sparse representations with a majority of tokens outside of the original query and document.
  arXiv  Detail & Related papers  (2023-07-19T22:48:02Z)
• Binary Embedding-based Retrieval at Tencent [30.44247353560061]
  Large-scale embedding-based retrieval (EBR) is the cornerstone of search-related industrial applications. We propose a binary embedding-based retrieval engine equipped with a recurrent binarization algorithm that enables customized bits per dimension. We successfully employed the introduced BEBR to Tencent products, including Sogou, Tencent Video, QQ World, etc.
  arXiv  Detail & Related papers  (2023-02-17T06:10:02Z)
• Precise Zero-Shot Dense Retrieval without Relevance Labels [60.457378374671656]
  Hypothetical Document Embeddings(HyDE) is a zero-shot dense retrieval system. We show that HyDE significantly outperforms the state-of-the-art unsupervised dense retriever Contriever.
  arXiv  Detail & Related papers  (2022-12-20T18:09:52Z)
• Rapid Person Re-Identification via Sub-space Consistency Regularization [51.76876061721556]
  Person Re-Identification (ReID) matches pedestrians across disjoint cameras. Existing ReID methods adopting real-value feature descriptors have achieved high accuracy, but they are low in efficiency due to the slow Euclidean distance computation. We propose a novel Sub-space Consistency Regularization (SCR) algorithm that can speed up the ReID procedure by 0.25$ times.
  arXiv  Detail & Related papers  (2022-07-13T02:44:05Z)
• Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval [41.93637130875804]
  Ad-hoc search calls for the selection of appropriate answers from a massive-scale corpus. Deep learning based document representation and ANN search techniques are allied to handle this task. A major challenge is that the ANN index can be too large to fit into memory, given the considerable size of answer corpus. In this work, we tackle this problem with Bi-Granular Document Representation, where the lightweight sparse embeddings are indexed and standby in memory for coarse-grained candidate search.
  arXiv  Detail & Related papers  (2022-01-14T12:02:47Z)
• Composably secure data processing for Gaussian-modulated continuous variable quantum key distribution [58.720142291102135]
  Continuous-variable quantum key distribution (QKD) employs the quadratures of a bosonic mode to establish a secret key between two remote parties. We consider a protocol with homodyne detection in the general setting of composable finite-size security. In particular, we analyze the high signal-to-noise regime which requires the use of high-rate (non-binary) low-density parity check codes.
  arXiv  Detail & Related papers  (2021-03-30T18:02:55Z)
• Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Verification [151.62491805851107]
  We develop $beta$-CROWN, a bound propagation based verifier that can fully encode per-neuron splits. $beta$-CROWN is close to three orders of magnitude faster than LP-based BaB methods for robustness verification. By terminating BaB early, our method can also be used for incomplete verification.
  arXiv  Detail & Related papers  (2021-03-11T11:56:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.