From Pixels to Words -- Towards Native One-Vision Models at Scale

• URL: http://arxiv.org/abs/2605.28820v1
• Date: Wed, 27 May 2026 17:59:55 GMT
• Title: From Pixels to Words -- Towards Native One-Vision Models at Scale
• Abstract summary: Current vision models (VLMs) typically stitch together separate image encoders and language decoders via multi-stage alignment.<n>We introduce NEO-ov, a native foundation model that learns cross-frame and pixel-word correspondence end-to-end.<n>NEO-ov largely narrows the gap to modular counterparts while excelling at fine-grained visual perception.
• Score: 122.87026673966544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current vision-language models (VLMs) typically stitch together separate image encoders and language decoders via multi-stage alignment, a modular framework that inevitably fragments pixel-level signals across frames and scatters early pixel-word interactions. In parallel, native VLMs, despite impressive performance on single images, remain largely unexplored in multi-image, video understanding, and spatial intelligence. Hence, we introduce NEO-ov, a native foundation model that learns cross-frame and pixel-word correspondence end-to-end, without any external encoders, auxiliary adapters, or post-hoc fusion. By eliminating module boundaries entirely, NEO-ov enables fine-grained and unified spatiotemporal modeling to emerge natively inside the model. Notably, NEO-ov largely narrows the gap to modular counterparts while excelling at fine-grained visual perception, validating that native "one-vision" architectures are not only feasible but competitive at scale. Beyond empirical performance, we unveil systematic architectural analyses and detailed training recipes to facilitate subsequent native multimodal modeling. Our code and models are publicly available at: https://github.com/EvolvingLMMs-Lab/NEO.

Related papers

• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation [108.23557570345356]
  Tuna-2 is a native unified multimodal model that performs visual understanding and generation directly based on pixel embeddings.<n>Experiments show that Tuna-2 achieves state-of-the-art performance in multimodal benchmarks.
  arXiv  Detail & Related papers  (2026-04-27T17:59:56Z)
• LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model [43.908068133557215]
  We present LLaDA2.0-Uni, a unified discrete diffusion large language model (dLLM) that supports multimodal understanding and generation.<n>Its architecture combines a fully semantic discrete tokenizer, a MoE-based dLLM backbone, and a diffusion decoder.
  arXiv  Detail & Related papers  (2026-04-22T17:20:42Z)
• From Pixels to Words -- Towards Native Vision-Language Primitives at Scale [77.93798335498703]
  We launch NEO, a novel family of native Vision-Language Models (VLMs) built from first principles.<n>NEO efficiently develops visual perception from scratch while mitigating vision-language conflicts inside a dense and monolithic model.<n>Our code and models are publicly available at: https://github.com/EvolvingLMMs-Lab/NEO.
  arXiv  Detail & Related papers  (2025-10-16T17:59:58Z)
• MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer [90.72238747690972]
  We present Manzano, a simple and scalable unified framework for multimodal large language models.<n>A single vision encoder feeds two adapters that produce continuous embeddings for image-to-text understanding and discrete tokens for text-to-image generation.<n>A unified autoregressive LLM predicts high-level semantics in the form of text and image tokens, with an auxiliary diffusion decoder subsequently translating the image tokens into pixels.
  arXiv  Detail & Related papers  (2025-09-19T17:58:00Z)
• Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space [9.327655601475605]
  We propose Nexus-Gen, a novel architecture that unifies image understanding, generation, and editing tasks in a shared image embedding space.<n>To mitigate the severe error accumulation during autoregressive embedding prediction, we propose a novel prefilled autoregression strategy.<n>Nexus-Gen achieves state-of-the-art performance on the evaluation benchmarks spanning image understanding, generation and editing tasks.
  arXiv  Detail & Related papers  (2025-04-30T06:30:48Z)
• Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment [6.614005142754584]
  Universal Sparse Autoencoders (USAEs) are a framework for uncovering and aligning interpretable concepts spanning multiple deep neural networks.<n>USAEs learn a universal concept space that can reconstruct and interpret the internal activations of multiple models at once.
  arXiv  Detail & Related papers  (2025-02-06T02:06:16Z)
• Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment [16.733970553781887]
  Recent findings suggest high semantic similarity between well-trained unimodal encoders.<n>We propose a novel framework that aligns vision and language using frozen unimodal encoders.
  arXiv  Detail & Related papers  (2024-09-28T17:57:32Z)
• Mono-ViFI: A Unified Learning Framework for Self-supervised Single- and Multi-frame Monocular Depth Estimation [11.611045114232187]
  Recent methods only conduct view synthesis between existing camera views, leading to insufficient guidance. We try to synthesize more virtual camera views by flow-based video frame making (VFI) For multi-frame inference, to sidestep the problem of dynamic objects encountered by explicit geometry-based methods like ManyDepth, we return to the feature fusion paradigm. We construct a unified self-supervised learning framework, named Mono-ViFI, to bilaterally connect single- and multi-frame depth.
  arXiv  Detail & Related papers  (2024-07-19T08:51:51Z)
• Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions [94.03133100056372]
  Moonshot is a new video generation model that conditions simultaneously on multimodal inputs of image and text. Model can be easily repurposed for a variety of generative applications, such as personalized video generation, image animation and video editing.
  arXiv  Detail & Related papers  (2024-01-03T16:43:47Z)
• Language Models are General-Purpose Interfaces [109.45478241369655]
  We propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language) We propose a semi-causal language modeling objective to jointly pretrain the interface and the modular encoders.
  arXiv  Detail & Related papers  (2022-06-13T17:34:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.