OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference

• URL: http://arxiv.org/abs/2412.11475v2
• Date: Wed, 25 Dec 2024 01:15:29 GMT
• Title: OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference
• Abstract summary: We present OmniVLM, a vision-language model for efficient on-device inference.<n>It reduces visual token sequence length from 729 to 81 tokens, significantly reducing computational overhead.<n>It outperforms existing baselines like nanoLLAVA within a 968M- parameter footprint.
• Score: 9.78611123915888
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present OmniVLM, a sub-billion-parameter vision-language model for efficient on-device inference. OmniVLM introduces a token compression mechanism that reduces visual token sequence length from 729 to 81 tokens, significantly reducing computational overhead while preserving visual-semantic fidelity. Through a multi-stage training pipeline of pretraining, supervised fine-tuning, and minimal-edit Direct Preference Optimization (DPO), OmniVLM matches the performance of larger models. On multiple benchmarks including ScienceQA, POPE, and MMMU, OmniVLM outperforms existing baselines like nanoLLAVA within a 968M-parameter footprint. Empirical results on the same laptop demonstrate 9.1x faster time-to-first-token (0.75s vs 6.82s) and 1.5x higher decoding speed (29.41 vs 19.20 tokens/s) compared to nanoLLAVA, enabling efficient deployment on edge devices. The model weights can be accessed on huggingface: \url{https://huggingface.co/NexaAIDev/OmniVLM-968M}, and the inference examples can be find in Appendix B.

Related papers

• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• DAM-VLA: Decoupled Asynchronous Multimodal Vision Language Action model [6.247270890079242]
  Vision-language-action (VLA) models inherit a shared synchronous clock from vision-language pretraining, processing every input at one rate.<n>A synchronous VLA oversamples slow modalities, undersamples fast ones, and caps action generation at the lowest effective frequency.<n>We present DAM-VLA, which maintains per-modality latent buffers refreshed at sensor rates and read continuously by the action head.
  arXiv  Detail & Related papers  (2026-06-10T13:59:07Z)
• OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models [42.615226139135174]
  We propose OmniSIFT, a token compression framework tailored for Omni-LLMs.<n>The entire framework is optimized end-to-end via a differentiable straight-through estimator.<n>With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.
  arXiv  Detail & Related papers  (2026-02-04T17:51:05Z)
• Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs [111.69640966866059]
  Sparse large language models (LLMs) with Mixture of Experts (MoE) and close to a trillion parameters are dominating the realm of most capable language models.<n>In this paper, we aim to uncover a recipe to harness such scale on Ascend NPUs.<n>The key goals are better usage of the computing resources under the dynamic sparse model structures and materializing the expected performance gain on the actual hardware.
  arXiv  Detail & Related papers  (2025-05-07T15:46:36Z)
• Mobile-VideoGPT: Fast and Accurate Video Understanding Language Model [60.171601995737646]
  Mobile-VideoGPT is an efficient multimodal framework for video understanding. It consists of lightweight dual visual encoders, efficient projectors, and a small language model (SLM) Our results show that Mobile-VideoGPT-0.5B can generate up to 46 tokens per second.
  arXiv  Detail & Related papers  (2025-03-27T17:59:58Z)
• Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models [50.214593234229255]
  We introduce the novel task of Extreme Short Token Reduction, which aims to represent entire videos using a minimal set of discrete tokens.<n>On the Extreme Short Token Reduction task, our VQToken compresses sequences to just 0.07 percent of their original length while incurring only a 0.66 percent drop in accuracy on the NextQA-MC benchmark.
  arXiv  Detail & Related papers  (2025-03-21T09:46:31Z)
• OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models [36.0400717590138]
  We present OmniMamba, the first linear-architecture-based multimodal generation model. It generates both text and images through a unified next-token prediction paradigm. It achieves competitive performance with JanusFlow while surpassing Show-o across benchmarks.
  arXiv  Detail & Related papers  (2025-03-11T17:59:46Z)
• UniTok: A Unified Tokenizer for Visual Generation and Understanding [69.09699034036124]
  Visual generative and understanding models typically rely on distinct tokenizers to process images.<n>We introduce UniTok, a unified tokenizer featuring a novel multi-codebook quantization mechanism.<n>In terms of final performance, UniTok sets a new record of 0.38 rFID and 78.6% zero-shot accuracy on ImageNet.
  arXiv  Detail & Related papers  (2025-02-27T17:47:01Z)
• MiniMax-01: Scaling Foundation Models with Lightning Attention [59.38940023647236]
  MiniMax-Text-01 and MiniMax-VL-01 offer superior capabilities in processing longer contexts. MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens.
  arXiv  Detail & Related papers  (2025-01-14T18:50:05Z)
• ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models [32.6661928486072]
  ATP-LLaVA is a novel approach that adaptively determines instance-specific token pruning ratios for each Large Language Model layer.<n>Our approach reduces the average token count by 75% while maintaining performance, with only a minimal 1.9% degradation across seven widely used benchmarks.
  arXiv  Detail & Related papers  (2024-11-30T11:42:35Z)
• Inference Optimal VLMs Need Only One Visual Token but Larger Models [54.01228554126122]
  Vision Language Models (VLMs) have demonstrated strong capabilities across various visual understanding and reasoning tasks. VLMs are often constrained by high latency during inference due to substantial compute required to process the large number of input tokens. We take some initial steps towards building approaches tailored for high token compression settings.
  arXiv  Detail & Related papers  (2024-11-05T18:54:21Z)
• TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices [36.714057078457195]
  We present TPI-LLM, a compute- and memory-efficient tensor parallel inference system for 70B-scale models. TPI-LLM keeps sensitive raw data local in the users' devices and introduces a sliding window memory scheduler. We show that TPI-LLM demonstrated over 80% less time-to-first-token and token latency compared to Accelerate.
  arXiv  Detail & Related papers  (2024-10-01T09:18:56Z)
• Token Turing Machines are Efficient Vision Models [2.8674240660437778]
  We propose Vision Token Turing Machines (ViTTM), an efficient, low-latency, memory-augmented Vision Transformer (ViT) ViTTMs are designed for non-sequential computer vision tasks such as image classification and segmentation.
  arXiv  Detail & Related papers  (2024-09-11T20:50:41Z)
• Matryoshka Query Transformer for Large Vision-Language Models [103.84600181927884]
  We introduce the Matryoshka Query Transformer (MQT), capable of encoding an image into m visual tokens during inference. We train a single model once, and flexibly and drastically reduce the number of inference-time visual tokens. Our model, MQT-LLAVA, matches LLaVA-1.5 performance across 11 benchmarks using a maximum of 256 tokens instead of LLaVA's fixed 576.
  arXiv  Detail & Related papers  (2024-05-29T17:39:42Z)
• An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models [65.37846460916042]
  We find out that the attention computation over visual tokens is of extreme inefficiency in the deep layers of popular LVLMs. We introduce FastV, a versatile plug-and-play method designed to optimize computational efficiency.
  arXiv  Detail & Related papers  (2024-03-11T14:35:32Z)
• MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices [73.46317110474064]
  MobileVLM is a competent multimodal vision language model (MMVLM) targeted to run on mobile devices. It comprises a set of language models at the scale of 1.4B and 2.7B parameters, trained from scratch, a multimodal vision model that is pre-trained in the CLIP fashion.
  arXiv  Detail & Related papers  (2023-12-28T08:21:24Z)
• VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking [57.552798046137646]
  Video masked autoencoder (VideoMAE) is a scalable and general self-supervised pre-trainer for building video foundation models. We successfully train a video ViT model with a billion parameters, which achieves a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-03-29T14:28:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.