Jasper-Token-Compression-600M Technical Report

• URL: http://arxiv.org/abs/2511.14405v2
• Date: Wed, 19 Nov 2025 06:03:10 GMT
• Title: Jasper-Token-Compression-600M Technical Report
• Abstract summary: This report presents the training methodology and evaluation results of the open-source Jasper-Token-Compression-600M model.<n>A key innovation of our model is the introduction of a one-dimensional convolution-based token compression module.<n>Our model performs with higher efficiency than a traditional 0.6B model while achieving performance comparable to that of an 8B model.
• Score: 1.349950008899546
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This technical report presents the training methodology and evaluation results of the open-source Jasper-Token-Compression-600M model, released in November 2025. Building on previous distillation-based recipes from the English Stella and Jasper models, we successfully extend this approach to a bilingual (English and Chinese) domain, further enhancing model performance through the incorporation of contrastive learning. A key innovation of our model is the introduction of a one-dimensional convolution-based token compression module. We dynamically adjust the compression rate during training, enabling the model to learn more robust and efficient compressed text representations. By combining knowledge distillation with token compression techniques, we achieve significant improvements in both embedding quality and inference efficiency. Our model performs with higher efficiency than a traditional 0.6B model while achieving performance comparable to that of an 8B model. For more information on the model release, visit: https://huggingface.co/infgrad/Jasper-Token-Compression-600M.

Related papers

• Mixture-of-Experts-based Entropy Model for Learned Image Compression [53.71337604556311]
  We propose a MoE-based Entropy model (MoEE) for learned image compression.<n>Our model achieves a BD-Rate improvement over VVC of -16.85% on the Kodak dataset.
  arXiv  Detail & Related papers  (2026-08-11T14:17:39Z)
• Model compression using knowledge distillation with integrated gradients [0.0]
  We introduce a novel method enhancing knowledge distillation with integrated gradients (IG)<n>Our approach overlays IG maps onto input images during training, providing student models with deeper insights into teacher models' decision-making processes.<n>Our method precomputes IG maps before training, transforming substantial runtime costs into a one-time preprocessing step.
  arXiv  Detail & Related papers  (2025-06-17T12:00:23Z)
• Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates [20.16951333751427]
  This paper presents a novel approach for speech foundation models compression that tightly integrates model pruning and parameter update into a single stage.<n> Experiments conducted on the LibriSpeech-100hr corpus suggest that our approach reduces the number of parameters of wav2vec2.0-base and HuBERT-large models by 65% and 60% respectively.
  arXiv  Detail & Related papers  (2025-05-28T17:24:21Z)
• Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruning [76.88243649182886]
  Hybrid architectures that combine Attention and State Space Models (SSMs) achieve state-of-the-art accuracy and runtime performance.<n>Recent work has demonstrated that applying compression and distillation to Attention-only models yields smaller, more accurate models at a fraction of the training cost.<n>We introduce a novel group-aware pruning strategy that preserves the structural integrity of SSM blocks and their sequence modeling capabilities.
  arXiv  Detail & Related papers  (2025-04-15T17:26:29Z)
• Compact Language Models via Pruning and Knowledge Distillation [61.56557874432008]
  Minitron models exhibit up to a 16% improvement in MMLU scores compared to training from scratch. Deriving 8B and 4B models from an already pretrained 15B model using our approach requires up to 40x fewer training tokens per model compared to training from scratch.
  arXiv  Detail & Related papers  (2024-07-19T21:47:57Z)
• Activations and Gradients Compression for Model-Parallel Training [85.99744701008802]
  We study how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We find that gradients require milder compression rates than activations. Experiments also show that models trained with TopK perform well only when compression is also applied during inference.
  arXiv  Detail & Related papers  (2024-01-15T15:54:54Z)
• Co-training and Co-distillation for Quality Improvement and Compression of Language Models [88.94539115180919]
  Knowledge Distillation (KD) compresses expensive pre-trained language models (PLMs) by transferring their knowledge to smaller models. Most smaller models fail to surpass the performance of the original larger model, resulting in sacrificing performance to improve inference speed. We propose Co-Training and Co-Distillation (CTCD), a novel framework that improves performance and inference speed together by co-training two models.
  arXiv  Detail & Related papers  (2023-11-06T03:29:00Z)
• Does compressing activations help model parallel training? [64.59298055364336]
  We present the first empirical study on the effectiveness of compression methods for model parallelism. We implement and evaluate three common classes of compression algorithms. We evaluate these methods across more than 160 settings and 8 popular datasets.
  arXiv  Detail & Related papers  (2023-01-06T18:58:09Z)
• Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition [7.450574974954803]
  Small memory bandwidth in smart devices prompts development of smaller Automatic Speech Recognition (ASR) models. Knowledge distillation (KD) is a popular model compression approach that has shown to achieve smaller model size. We propose a multi-stage progressive approach to compress the conformer transducer model using KD.
  arXiv  Detail & Related papers  (2022-10-01T02:23:00Z)
• MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation [68.30497162547768]
  We propose MoEBERT, which uses a Mixture-of-Experts structure to increase model capacity and inference speed. We validate the efficiency and effectiveness of MoEBERT on natural language understanding and question answering tasks.
  arXiv  Detail & Related papers  (2022-04-15T23:19:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.