VLF-MSC: Vision-Language Feature-Based Multimodal Semantic Communication System

• URL: http://arxiv.org/abs/2511.10074v1
• Date: Fri, 14 Nov 2025 01:30:32 GMT
• Title: VLF-MSC: Vision-Language Feature-Based Multimodal Semantic Communication System
• Authors: Gwangyeon Ahn, Jiwan Seo, Joonhyuk Kang,
• Abstract summary: Vision-Language Feature-based Multimodal Semantic Communication (VLF-MSC) is a unified system that transmits a single vision-language representation to support both image and text generation at the receiver.<n>By leveraging foundation models, the system achieves robustness to channel noise while preserving semantic fidelity.
• Score: 0.9176056742068811
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose Vision-Language Feature-based Multimodal Semantic Communication (VLF-MSC), a unified system that transmits a single compact vision-language representation to support both image and text generation at the receiver. Unlike existing semantic communication techniques that process each modality separately, VLF-MSC employs a pre-trained vision-language model (VLM) to encode the source image into a vision-language semantic feature (VLF), which is transmitted over the wireless channel. At the receiver, a decoder-based language model and a diffusion-based image generator are both conditioned on the VLF to produce a descriptive text and a semantically aligned image. This unified representation eliminates the need for modality-specific streams or retransmissions, improving spectral efficiency and adaptability. By leveraging foundation models, the system achieves robustness to channel noise while preserving semantic fidelity. Experiments demonstrate that VLF-MSC outperforms text-only and image-only baselines, achieving higher semantic accuracy for both modalities under low SNR with significantly reduced bandwidth.

Related papers

• Context Video Semantic Transmission with Variable Length and Rate Coding over MIMO Channels [49.624608869195065]
  We propose the context video semantic transmission (CVST) framework for wireless video transmission.<n>We learn a context-channel correlation map to explicitly formulate the relationships between feature groups and multiple input multiple output (MIMO) subchannels.<n>We demonstrate substantial performance gains over various standardized separated coding methods and recent wireless video semantic communication approaches.
  arXiv  Detail & Related papers  (2025-12-23T10:48:43Z)
• WVSC: Wireless Video Semantic Communication with Multi-frame Compensation [56.63352157833874]
  Existing wireless video transmission schemes directly conduct video coding in pixel level.<n>We propose a wireless video semantic communication framework, abbreviated as WVSC, which integrates the idea of semantic communication into wireless video transmission scenarios.
  arXiv  Detail & Related papers  (2025-03-27T06:27:15Z)
• Vision Transformer Based Semantic Communications for Next Generation Wireless Networks [3.8095664680229935]
  This paper presents a Vision Transformer (ViT)-based semantic communication framework.<n>By equipping ViT as the encoder-decoder framework, the proposed architecture can proficiently encode images into a high semantic content.<n>The architecture based on the proposed ViT network achieves the Peak Signal-versato-noise Ratio (PSNR) of 38 dB.
  arXiv  Detail & Related papers  (2025-03-21T16:23:02Z)
• ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification [52.405499816861635]
  Multiple instance learning (MIL)-based framework has become the mainstream for processing the whole slide image (WSI)<n>We propose a dual-scale vision-language multiple instance learning (ViLa-MIL) framework for whole slide image classification.
  arXiv  Detail & Related papers  (2025-02-12T13:28:46Z)
• Vision Transformer-based Semantic Communications With Importance-Aware Quantization [13.328970689723096]
  This paper presents a vision transformer (ViT)-based semantic communication system with importance-aware quantization (IAQ) for wireless image transmission.<n>We show that our IAQ framework outperforms conventional image compression methods in both error-free and realistic communication scenarios.
  arXiv  Detail & Related papers  (2024-12-08T19:24:47Z)
• Visual Language Model based Cross-modal Semantic Communication Systems [42.321208020228894]
  We propose a novel Vision-Language Model-based Cross-modal Semantic Communication system. The VLM-CSC comprises three novel components. The experimental simulations validate the effectiveness, adaptability, and robustness of the CSC system.
  arXiv  Detail & Related papers  (2024-05-06T08:59:16Z)
• Communication-Efficient Framework for Distributed Image Semantic Wireless Transmission [68.69108124451263]
  Federated learning-based semantic communication (FLSC) framework for multi-task distributed image transmission with IoT devices. Each link is composed of a hierarchical vision transformer (HVT)-based extractor and a task-adaptive translator. Channel state information-based multiple-input multiple-output transmission module designed to combat channel fading and noise.
  arXiv  Detail & Related papers  (2023-08-07T16:32:14Z)
• OmniVL:One Foundation Model for Image-Language and Video-Language Tasks [117.57580168859512]
  We present OmniVL, a new foundation model to support both image-language and video-language tasks using one universal architecture. We demonstrate, for the first time, such a paradigm benefits both image and video tasks, as opposed to the conventional one-directional transfer. We introduce a novel unified vision-language contrastive (UniVLC) loss to leverage image-text, video-text, image-label (e.g., image classification), video-label (e.g., video action recognition) data together.
  arXiv  Detail & Related papers  (2022-09-15T17:59:59Z)
• Semantic Image Synthesis via Diffusion Models [174.24523061460704]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks.<n>Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches.<n>We propose a novel framework based on DDPM for semantic image synthesis.
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.