Fast, memory-efficient genomic interval tokenizers for modern machine learning

• URL: http://arxiv.org/abs/2511.01555v1
• Date: Mon, 03 Nov 2025 13:18:36 GMT
• Title: Fast, memory-efficient genomic interval tokenizers for modern machine learning
• Abstract summary: gtars-tokenizers is a library that maps genomic intervals to a predefined universe or vocabulary of regions.<n> gtars-tokenizers implements two overlap methods (BITS and AIList) and integrates seamlessly with modern ML frameworks.<n>This token-based approach bridges genomics and machine learning, supporting scalable analysis of interval data across diverse environments.
• Score: 11.571506663357015
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Introduction: Epigenomic datasets from high-throughput sequencing experiments are commonly summarized as genomic intervals. As the volume of this data grows, so does interest in analyzing it through deep learning. However, the heterogeneity of genomic interval data, where each dataset defines its own regions, creates barriers for machine learning methods that require consistent, discrete vocabularies. Methods: We introduce gtars-tokenizers, a high-performance library that maps genomic intervals to a predefined universe or vocabulary of regions, analogous to text tokenization in natural language processing. Built in Rust with bindings for Python, R, CLI, and WebAssembly, gtars-tokenizers implements two overlap methods (BITS and AIList) and integrates seamlessly with modern ML frameworks through Hugging Face-compatible APIs. Results: The gtars-tokenizers package achieves top efficiency for large-scale datasets, while enabling genomic intervals to be processed using standard ML workflows in PyTorch and TensorFlow without ad hoc preprocessing. This token-based approach bridges genomics and machine learning, supporting scalable and standardized analysis of interval data across diverse computational environments. Availability: PyPI and GitHub: https://github.com/databio/gtars.

Related papers

• xaitimesynth: A Python Package for Evaluating Attribution Methods for Time Series with Synthetic Ground Truth [0.0]
  We introduce xaitimesynth, a Python package that provides reusable infrastructure for time series evaluation.<n>xaitimesynth generates synthetic time series following an additive model.<n>A fluent data generation API and YAML configuration format allow flexible and reproducible dataset definitions.
  arXiv  Detail & Related papers  (2026-03-06T18:45:24Z)
• MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging [65.07273789940116]
  This paper introduces a hierarchical architecture that jointly optimize a dynamic genomic tokenizer and latent Transformers with context-aware pre-training tasks.<n> MergeDNA achieves superior performance on three popular DNA benchmarks and several multi-omics tasks with fine-tuning or zero-shot evaluation.
  arXiv  Detail & Related papers  (2025-11-17T19:27:41Z)
• CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training [63.07024608399447]
  We propose an automated framework that discovers, evaluates, and refines data mixtures in a pre-training setting.<n>We introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20 clusters as a research playground, and ClimbMix, a compact yet powerful 400-billion-token dataset.
  arXiv  Detail & Related papers  (2025-04-17T17:58:13Z)
• SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation [55.2480439325792]
  We study the synthesis of six datasets, covering topic classification, sentiment analysis, tone detection, and humor. We find that SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance.
  arXiv  Detail & Related papers  (2024-05-16T12:22:41Z)
• Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Boosted Trees [11.732842929815401]
  Tabular data is hard to acquire and is subject to missing values. This paper introduces a novel approach for generating and imputing mixed-type (continuous and categorical) data. In contrast to prior methods that rely on neural networks to learn the score function or the vector field, we adopt XGBoost.
  arXiv  Detail & Related papers  (2023-09-18T17:49:09Z)
• PyPOTS: A Python Toolkit for Machine Learning on Partially-Observed Time Series [20.491714178518155]
  PyPOTS is an open-source library for data mining and analysis.<n>It provides easy access to diverse algorithms categorized into five tasks.<n>PyPOTS is available on PyPI, Anaconda, and Docker.
  arXiv  Detail & Related papers  (2023-05-30T07:57:05Z)
• pyGSL: A Graph Structure Learning Toolkit [14.000763778781547]
  pyGSL is a Python library that provides efficient implementations of state-of-the-art graph structure learning models. pyGSL is written in GPU-friendly ways, allowing one to scale to much larger network tasks.
  arXiv  Detail & Related papers  (2022-11-07T14:23:10Z)
• PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks [68.96484488899901]
  We present PARTIME, a library designed to speed up neural networks whenever data is continuously streamed over time. PARTIME starts processing each data sample at the time in which it becomes available from the stream. Experiments are performed in order to empirically compare PARTIME with classic non-parallel neural computations in online learning.
  arXiv  Detail & Related papers  (2022-10-17T14:49:14Z)
• On the Effectiveness of Pretrained Models for API Learning [8.788509467038743]
  Developers frequently use APIs to implement certain functionalities, such as parsing Excel Files, reading and writing text files line by line, etc. Developers can greatly benefit from automatic API usage sequence generation based on natural language queries for building applications in a faster and cleaner manner. Existing approaches utilize information retrieval models to search for matching API sequences given a query or use RNN-based encoder-decoder to generate API sequences.
  arXiv  Detail & Related papers  (2022-04-05T20:33:24Z)
• SapientML: Synthesizing Machine Learning Pipelines by Learning from Human-Written Solutions [28.718446733713183]
  We propose an AutoML SapientML that can learn from a corpus of existing datasets and their human-written pipelines. We have created a training corpus of 1094 pipelines spanning 170 datasets, and evaluated SapientML on a set of 41 benchmark datasets. Our evaluation shows that SapientML produces the best or comparable accuracy on 27 of the benchmarks while the second best tool fails to even produce a pipeline on 9 of the instances.
  arXiv  Detail & Related papers  (2022-02-18T20:45:47Z)
• Cherry-Picking Gradients: Learning Low-Rank Embeddings of Visual Data via Differentiable Cross-Approximation [53.95297550117153]
  We propose an end-to-end trainable framework that processes large-scale visual data tensors by looking emphat a fraction of their entries only. The proposed approach is particularly useful for large-scale multidimensional grid data, and for tasks that require context over a large receptive field.
  arXiv  Detail & Related papers  (2021-05-29T08:39:57Z)
• Open Graph Benchmark: Datasets for Machine Learning on Graphs [86.96887552203479]
  We present the Open Graph Benchmark (OGB) to facilitate scalable, robust, and reproducible graph machine learning (ML) research. OGB datasets are large-scale, encompass multiple important graph ML tasks, and cover a diverse range of domains. For each dataset, we provide a unified evaluation protocol using meaningful application-specific data splits and evaluation metrics.
  arXiv  Detail & Related papers  (2020-05-02T03:09:50Z)
• giotto-tda: A Topological Data Analysis Toolkit for Machine Learning and Data Exploration [4.8353738137338755]
  giotto-tda is a Python library that integrates high-performance topological data analysis with machine learning. The library's ability to handle various types of data is rooted in a wide range of preprocessing techniques.
  arXiv  Detail & Related papers  (2020-04-06T10:53:57Z)
• Multi-layer Optimizations for End-to-End Data Analytics [71.05611866288196]
  We introduce Iterative Functional Aggregate Queries (IFAQ), a framework that realizes an alternative approach. IFAQ treats the feature extraction query and the learning task as one program given in the IFAQ's domain-specific language. We show that a Scala implementation of IFAQ can outperform mlpack, Scikit, and specialization by several orders of magnitude for linear regression and regression tree models over several relational datasets.
  arXiv  Detail & Related papers  (2020-01-10T16:14:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.