Large AI Model-Based Semantic Communications
- URL: http://arxiv.org/abs/2307.03492v1
- Date: Fri, 7 Jul 2023 10:01:08 GMT
- Title: Large AI Model-Based Semantic Communications
- Authors: Feibo Jiang, Yubo Peng, Li Dong, Kezhi Wang, Kun Yang, Cunhua Pan,
Xiaohu You
- Abstract summary: We propose a large AI model-based SC framework (LAM-SC) specifically designed for image data.
We first design the segment anything model (SAM)-based KB (SKB) that can split the original image into different semantic segments by universal semantic knowledge.
We then present an attention-based semantic integration (ASI) to weigh the semantic segments generated by SKB without human participation and integrate them as the semantic-aware image.
- Score: 58.394527592974576
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Semantic communication (SC) is an emerging intelligent paradigm, offering
solutions for various future applications like metaverse, mixed-reality, and
the Internet of everything. However, in current SC systems, the construction of
the knowledge base (KB) faces several issues, including limited knowledge
representation, frequent knowledge updates, and insecure knowledge sharing.
Fortunately, the development of the large AI model provides new solutions to
overcome above issues. Here, we propose a large AI model-based SC framework
(LAM-SC) specifically designed for image data, where we first design the
segment anything model (SAM)-based KB (SKB) that can split the original image
into different semantic segments by universal semantic knowledge. Then, we
present an attention-based semantic integration (ASI) to weigh the semantic
segments generated by SKB without human participation and integrate them as the
semantic-aware image. Additionally, we propose an adaptive semantic compression
(ASC) encoding to remove redundant information in semantic features, thereby
reducing communication overhead. Finally, through simulations, we demonstrate
the effectiveness of the LAM-SC framework and the significance of the large AI
model-based KB development in future SC paradigms.
Related papers
- Weakly-supervised Semantic Segmentation via Dual-stream Contrastive Learning of Cross-image Contextual Information [10.77139542242678]
Weakly supervised semantic segmentation (WSSS) aims at learning a semantic segmentation model with only image-level tags.
Most current WSSS methods focus on a limited single image (pixel-wise) information while ignoring the valuable inter-image (semantic-wise) information.
arXiv Detail & Related papers (2024-05-08T09:35:26Z) - Large Multi-modality Model Assisted AI-Generated Image Quality Assessment [53.182136445844904]
We introduce a large Multi-modality model Assisted AI-Generated Image Quality Assessment (MA-AGIQA) model.
It uses semantically informed guidance to sense semantic information and extract semantic vectors through carefully designed text prompts.
It achieves state-of-the-art performance, and demonstrates its superior generalization capabilities on assessing the quality of AI-generated images.
arXiv Detail & Related papers (2024-04-27T02:40:36Z) - Agent-driven Generative Semantic Communication with Cross-Modality and Prediction [57.335922373309074]
We propose a novel agent-driven generative semantic communication framework based on reinforcement learning.
In this work, we develop an agent-assisted semantic encoder with cross-modality capability, which can track the semantic changes, channel condition, to perform adaptive semantic extraction and sampling.
The effectiveness of the designed models has been verified using the UA-DETRAC dataset, demonstrating the performance gains of the overall A-GSC framework.
arXiv Detail & Related papers (2024-04-10T13:24:27Z) - Large AI Model Empowered Multimodal Semantic Communications [51.17527319441436]
We propose a Large AI Model-based Multimodal SC (LAM-MSC) framework.
We first present the SC-based Multimodal Alignment (MMA)
Then, a personalized LLM-based Knowledge Base (LKB) is proposed.
Finally, we apply the Conditional Generative adversarial networks-based channel Estimation (CGE) to obtain Channel State Information (CSI)
arXiv Detail & Related papers (2023-09-03T19:24:34Z) - Semantic Communications for Artificial Intelligence Generated Content
(AIGC) Toward Effective Content Creation [75.73229320559996]
This paper develops a conceptual model for the integration of AIGC and SemCom.
A novel framework that employs AIGC technology is proposed as an encoder and decoder for semantic information.
The framework can adapt to different types of content generated, the required quality, and the semantic information utilized.
arXiv Detail & Related papers (2023-08-09T13:17:21Z) - Segment Anything Meets Semantic Communication [15.183506390391988]
This paper explores the application of foundation models, particularly the Segment Anything Model (SAM) developed by Meta AI Research, to improve semantic communication.
By employing SAM's segmentation capability and lightweight neural network architecture for semantic coding, we propose a practical approach to semantic communication.
arXiv Detail & Related papers (2023-06-03T11:54:56Z) - Causal Semantic Communication for Digital Twins: A Generalizable
Imitation Learning Approach [74.25870052841226]
A digital twin (DT) leverages a virtual representation of the physical world, along with communication (e.g., 6G), computing, and artificial intelligence (AI) technologies to enable many connected intelligence services.
Wireless systems can exploit the paradigm of semantic communication (SC) for facilitating informed decision-making under strict communication constraints.
A novel framework called causal semantic communication (CSC) is proposed for DT-based wireless systems.
arXiv Detail & Related papers (2023-04-25T00:15:00Z) - Wireless Resource Management in Intelligent Semantic Communication
Networks [15.613654766345702]
We address the user association (UA) and bandwidth allocation problems in an ISC-enabled heterogeneous network (ISC-HetNet)
We propose a two-stage solution, including a programming method to obtain an objective, and a algorithm in the second stage to reach the optimality of UA and BA.
arXiv Detail & Related papers (2022-02-15T18:28:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.