TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications

• URL: http://arxiv.org/abs/2603.00482v1
• Date: Sat, 28 Feb 2026 05:52:35 GMT
• Title: TokenCom: Vision-Language Model for Multimodal and Multitask Token Communications
• Abstract summary: TaiChi is a novel VLM framework designed for token communications.<n>A Bilateral Attention Network (BAN) is introduced to intelligently fuse multi-scale visual tokens.<n>A Kolmogorov Arnold Network (KAN)-based modality projector with learnable activation functions is employed to achieve precise nonlinear alignment.
• Score: 83.27113284973684
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual-Language Models (VLMs), with their strong capabilities in image and text understanding, offer a solid foundation for intelligent communications. However, their effectiveness is constrained by limited token granularity, overlong visual token sequences, and inadequate cross-modal alignment. To overcome these challenges, we propose TaiChi, a novel VLM framework designed for token communications. TaiChi adopts a dual-visual tokenizer architecture that processes both high- and low-resolution images to collaboratively capture pixel-level details and global conceptual features. A Bilateral Attention Network (BAN) is introduced to intelligently fuse multi-scale visual tokens, thereby enhancing visual understanding and producing compact visual tokens. In addition, a Kolmogorov Arnold Network (KAN)-based modality projector with learnable activation functions is employed to achieve precise nonlinear alignment from visual features to the text semantic space, thus minimizing information loss. Finally, TaiChi is integrated into a multimodal and multitask token communication system equipped with a joint VLM-channel coding scheme. Experimental results validate the superior performance of TaiChi, as well as the feasibility and effectiveness of the TaiChi-driven token communication system.

Related papers

• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation [68.03746493619285]
  We present JoyAI-Image, a unified multimodal foundation model for visual understanding, text-to-image generation, and instruction-guided image editing.<n>We build a scalable training recipe that combines unified instruction tuning, long-text rendering supervision, spatially grounded data, and both general and spatial editing signals.<n> Experiments across understanding, generation, long-text rendering, and editing benchmarks show that JoyAI-Image achieves state-of-the-art or highly competitive performance.
  arXiv  Detail & Related papers  (2026-05-05T15:49:47Z)
• CASA: Cross-Attention via Self-Attention for Efficient Vision-Language Fusion [30.426836071099885]
  Vision-language models (VLMs) are commonly trained by inserting image tokens from a pretrained vision encoder into the textual stream of a language model.<n>This allows text and image information to fully attend to one another within the model, but becomes extremely costly for high-resolution images, long conversations, or streaming videos, both in memory and compute.<n>We propose CASA, Cross-Attention via Self-Attention, a simple and efficient paradigm which substantially reduces the gap with full token insertion on common image understanding benchmarks.
  arXiv  Detail & Related papers  (2025-12-22T16:21:39Z)
• Unified Multimodal Understanding via Byte-Pair Visual Encoding [34.96534298857146]
  Multimodal large language models (MLLMs) have made significant progress in vision-language understanding.<n>We present a framework that unifies multimodal understanding by applying byte-pair encoding to visual tokens.
  arXiv  Detail & Related papers  (2025-06-30T09:08:08Z)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations [33.11867433769496]
  This paper presents a framework that attempts to unify visual understanding and generation within a shared semantic representation.<n>At its core is the Text-Aligned Tokenizer (TA-Tok), which converts images into discrete tokens using a text-aligned codebook projected from a large language model's (LLM) vocabulary.<n> Experiments across benchmarks show that Tar matches or surpasses existing multimodal LLM methods, achieving faster convergence and greater training efficiency.
  arXiv  Detail & Related papers  (2025-06-23T17:59:14Z)
• MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model [49.931663904599205]
  MaVEn is an innovative framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. We show that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.
  arXiv  Detail & Related papers  (2024-08-22T11:57:16Z)
• Towards Semantic Equivalence of Tokenization in Multimodal LLM [149.11720372278273]
  Vision tokenization is essential for semantic alignment between vision and language.<n>This paper proposes a novel dynamic Semantic-Equivalent Vision Tokenizer (SeTok)<n>SeTok groups visual features into semantic units via a dynamic clustering algorithm.<n>The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features.
  arXiv  Detail & Related papers  (2024-06-07T17:55:43Z)
• Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens [76.40196364163663]
  We propose a learning-based vision-language pre-training approach, such as CLIP. We show that our method can learn more comprehensive representations and capture meaningful cross-modal correspondence.
  arXiv  Detail & Related papers  (2023-03-27T00:58:39Z)
• mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections [104.14624185375897]
  mPLUG is a new vision-language foundation model for both cross-modal understanding and generation. It achieves state-of-the-art results on a wide range of vision-language downstream tasks, such as image captioning, image-text retrieval, visual grounding and visual question answering.
  arXiv  Detail & Related papers  (2022-05-24T11:52:06Z)
• Single-Stream Multi-Level Alignment for Vision-Language Pretraining [103.09776737512078]
  We propose a single stream model that aligns the modalities at multiple levels. We achieve this using two novel tasks: symmetric cross-modality reconstruction and a pseudo-labeled key word prediction. We demonstrate top performance on a set of Vision-Language downstream tasks such as zero-shot/fine-tuned image/text retrieval, referring expression, and VQA.
  arXiv  Detail & Related papers  (2022-03-27T21:16:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.