NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference

• URL: http://arxiv.org/abs/2609.01657v1
• Date: Mon, 31 Aug 2026 13:08:15 GMT
• Title: NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference
• Abstract summary: We introduce NeoMME, a family of 260M and 800M Multimodal and bidirectional Multilingual document encoders.<n>Both models are pretrained from scratch with a masked discrete-diffusion text objective.<n>On ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models with 0.523 nDCG@10.<n>At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT.
• Score: 1.3961509337714324
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.

Related papers

• Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model [52.79436545460808]
  We present daVinci-MagiHuman, an open-source audio-video generative foundation model for human-centric generation.<n>DaVinci-MagiHuman jointly generates synchronized video and audio using a single-stream Transformer.<n>It supports multilingual spoken generation across Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French.
  arXiv  Detail & Related papers  (2026-03-23T13:49:06Z)
• Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation [66.53544128707817]
  Cheers is a unified multimodal model that decouples patch-level details from semantic representations.<n> Cheers matches or surpasses advanced UMMs in both visual understanding and generation.
  arXiv  Detail & Related papers  (2026-03-13T08:55:27Z)
• Balancing Understanding and Generation in Discrete Diffusion Models [58.62235340638143]
  Masked Diffusion Language Models (MDLM) excel at semantic understanding and zero-shot generalization.<n>Uniform-noise Diffusion Language Models (UDLM) achieve strong few-step generation quality.<n>We propose XDLM, which bridges the two paradigms via a stationary noise kernel.
  arXiv  Detail & Related papers  (2026-02-01T18:00:35Z)
• Native LLM and MLLM Inference at Scale on Apple Silicon [0.8122270502556375]
  We present vllm-mlx, a framework for efficient LLM and MLLM inference on Apple Silicon built native on MLX.<n>For text models, we achieve 21% to 87% higher throughput than llama- across models ranging from Qwen3-0.6B to Nemotron-30B.<n>For multimodal models, we introduce content-based caching that eliminates redundant vision encoding by identifying identical images through content hashing, regardless of input format.
  arXiv  Detail & Related papers  (2026-01-27T03:11:02Z)
• Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models [92.18057318458528]
  Token-Shuffle is a novel method that reduces the number of image tokens in Transformer.<n>Our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis.<n>In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15.
  arXiv  Detail & Related papers  (2025-04-24T17:59:56Z)
• OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models [36.0400717590138]
  We present OmniMamba, the first linear-architecture-based multimodal generation model.<n>It generates both text and images through a unified next-token prediction paradigm.<n>It achieves competitive performance with JanusFlow while surpassing Show-o across benchmarks.
  arXiv  Detail & Related papers  (2025-03-11T17:59:46Z)
• Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling [4.293921765264583]
  Byte Language Models (BLMs) have emerged to overcome tokenization, yet the excessive length of bytestreams requires new architectural paradigms.<n>We present the Multiscale Byte Language Model (MBLM), a model-agnostic hierarchical decoder stack that allows training with context windows of $5$M bytes on single GPU in full model precision.
  arXiv  Detail & Related papers  (2025-02-20T13:31:50Z)
• CMC-Bench: Towards a New Paradigm of Visual Signal Compression [85.1839779884282]
  We introduce CMC-Bench, a benchmark of the cooperative performance of Image-to-Text (I2T) and Text-to-Image (T2I) models for image compression. At ultra-lows, this paper proves that the combination of some I2T and T2I models has surpassed the most advanced visual signal protocols.
  arXiv  Detail & Related papers  (2024-06-13T17:41:37Z)
• BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning [91.93547262073213]
  Vision-Language (VL) models with the Two-Tower architecture have dominated visual representation learning in recent years. We propose BridgeTower, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. BridgeTower achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data.
  arXiv  Detail & Related papers  (2022-06-17T09:42:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.