From Pixels to Words -- Towards Native Vision-Language Primitives at Scale

• URL: http://arxiv.org/abs/2510.14979v1
• Date: Thu, 16 Oct 2025 17:59:58 GMT
• Title: From Pixels to Words -- Towards Native Vision-Language Primitives at Scale
• Abstract summary: We launch NEO, a novel family of native Vision-Language Models (VLMs) built from first principles.<n>NEO efficiently develops visual perception from scratch while mitigating vision-language conflicts inside a dense and monolithic model.<n>Our code and models are publicly available at: https://github.com/EvolvingLMMs-Lab/NEO.
• Score: 77.93798335498703
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The edifice of native Vision-Language Models (VLMs) has emerged as a rising contender to typical modular VLMs, shaped by evolving model architectures and training paradigms. Yet, two lingering clouds cast shadows over its widespread exploration and promotion: (-) What fundamental constraints set native VLMs apart from modular ones, and to what extent can these barriers be overcome? (-) How to make research in native VLMs more accessible and democratized, thereby accelerating progress in the field. In this paper, we clarify these challenges and outline guiding principles for constructing native VLMs. Specifically, one native VLM primitive should: (i) effectively align pixel and word representations within a shared semantic space; (ii) seamlessly integrate the strengths of formerly separate vision and language modules; (iii) inherently embody various cross-modal properties that support unified vision-language encoding, aligning, and reasoning. Hence, we launch NEO, a novel family of native VLMs built from first principles, capable of rivaling top-tier modular counterparts across diverse real-world scenarios. With only 390M image-text examples, NEO efficiently develops visual perception from scratch while mitigating vision-language conflicts inside a dense and monolithic model crafted from our elaborate primitives. We position NEO as a cornerstone for scalable and powerful native VLMs, paired with a rich set of reusable components that foster a cost-effective and extensible ecosystem. Our code and models are publicly available at: https://github.com/EvolvingLMMs-Lab/NEO.

Related papers

• From Pixels to Words -- Towards Native One-Vision Models at Scale [122.87026673966544]
  Current vision models (VLMs) typically stitch together separate image encoders and language decoders via multi-stage alignment.<n>We introduce NEO-ov, a native foundation model that learns cross-frame and pixel-word correspondence end-to-end.<n>NEO-ov largely narrows the gap to modular counterparts while excelling at fine-grained visual perception.
  arXiv  Detail & Related papers  (2026-05-27T17:59:55Z)
• SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture [110.20462227888915]
  We introduce SenseNova-U1, a native unified multimodal paradigm built upon NEO-unify.<n>We show two native unified variants, SenseNova-U1-8B-MoT and SenseNova-U1-A3B-MoT, built on dense (8B) and mixture-of-experts (30B-A3B) understanding baselines.<n>We also show detailed model design, data preprocessing, pre-/post-training, and inference strategies to support community research.
  arXiv  Detail & Related papers  (2026-05-12T17:59:58Z)
• HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding [91.0552157725366]
  This paper presents a novel high-performance monolithic VLM named HoVLE.<n>It converts visual and textual inputs into a shared space, allowing LLMs to process images in the same way as texts.<n>Our experiments show that HoVLE achieves performance close to leading compositional models on various benchmarks.
  arXiv  Detail & Related papers  (2024-12-20T18:59:59Z)
• Liquid: Language Models are Scalable and Unified Multi-modal Generators [112.71734051183726]
  Liquid is an auto-regressive generation paradigm that seamlessly integrates visual comprehension and generation.<n>Unlike previous multimodal large language model (MLLM), Liquid achieves this integration using a single large language model.<n>For the first time, Liquid uncovers a scaling law that performance drop unavoidably brought by the unified training of visual and language tasks.
  arXiv  Detail & Related papers  (2024-12-05T16:48:16Z)
• An Introduction to Vision-Language Modeling [128.6223984157515]
  The vision-language model (VLM) applications will significantly impact our relationship with technology. We introduce what VLMs are, how they work, and how to train them. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos.
  arXiv  Detail & Related papers  (2024-05-27T15:01:23Z)
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks [92.03764152132315]
  We design a large-scale vision-language foundation model (InternVL), which scales up the vision foundation model to 6 billion parameters. This model can be broadly applied to and achieve state-of-the-art performance on 32 generic visual-linguistic benchmarks. It has powerful visual capabilities and can be a good alternative to the ViT-22B.
  arXiv  Detail & Related papers  (2023-12-21T18:59:31Z)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization [52.935150075484074]
  We introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously.
  arXiv  Detail & Related papers  (2023-09-09T03:01:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.