Segment Anything Meets Semantic Communication

• URL: http://arxiv.org/abs/2306.02094v1
• Date: Sat, 3 Jun 2023 11:54:56 GMT
• Title: Segment Anything Meets Semantic Communication
• Authors: Shehbaz Tariq, Brian Estadimas Arfeto, Chaoning Zhang, Hyundong Shin
• Abstract summary: This paper explores the application of foundation models, particularly the Segment Anything Model (SAM) developed by Meta AI Research, to improve semantic communication. By employing SAM's segmentation capability and lightweight neural network architecture for semantic coding, we propose a practical approach to semantic communication.
• Score: 15.183506390391988
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In light of the diminishing returns of traditional methods for enhancing transmission rates, the domain of semantic communication presents promising new frontiers. Focusing on image transmission, this paper explores the application of foundation models, particularly the Segment Anything Model (SAM) developed by Meta AI Research, to improve semantic communication. SAM is a promptable image segmentation model that has gained attention for its ability to perform zero-shot segmentation tasks without explicit training or domain-specific knowledge. By employing SAM's segmentation capability and lightweight neural network architecture for semantic coding, we propose a practical approach to semantic communication. We demonstrate that this approach retains critical semantic features, achieving higher image reconstruction quality and reducing communication overhead. This practical solution eliminates the resource-intensive stage of training a segmentation model and can be applied to any semantic coding architecture, paving the way for real-world applications.

Related papers

• Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning [86.99944014645322]
  We introduce a novel framework, Meta-Exploiting Frequency Prior for Cross-Domain Few-Shot Learning. We decompose each query image into its high-frequency and low-frequency components, and parallel incorporate them into the feature embedding network. Our framework establishes new state-of-the-art results on multiple cross-domain few-shot learning benchmarks.
  arXiv  Detail & Related papers  (2024-11-03T04:02:35Z)
• Trustworthy Image Semantic Communication with GenAI: Explainablity, Controllability, and Efficiency [59.15544887307901]
  Image semantic communication (ISC) has garnered significant attention for its potential to achieve high efficiency in visual content transmission. Existing ISC systems based on joint source-channel coding face challenges in interpretability, operability, and compatibility. We propose a novel trustworthy ISC framework that employs Generative Artificial Intelligence (GenAI) for multiple downstream inference tasks.
  arXiv  Detail & Related papers  (2024-08-07T14:32:36Z)
• Autoencoder-Based Domain Learning for Semantic Communication with Conceptual Spaces [1.7404865362620803]
  We develop a framework for learning a domain of a conceptual space model using only the raw data with high-level property labels. In experiments using the MNIST and CelebA datasets, we show that the domains learned using the framework maintain semantic similarity relations and possess interpretable dimensions.
  arXiv  Detail & Related papers  (2024-01-29T21:08:33Z)
• Self-guided Few-shot Semantic Segmentation for Remote Sensing Imagery Based on Large Vision Models [14.292149307183967]
  This research introduces a structured framework designed for the automation of few-shot semantic segmentation. It utilizes the SAM model and facilitates a more efficient generation of semantically discernible segmentation outcomes. Central to our methodology is a novel automatic prompt learning approach, leveraging prior guided masks to produce coarse pixel-wise prompts for SAM.
  arXiv  Detail & Related papers  (2023-11-22T07:07:55Z)
• Large AI Model-Based Semantic Communications [48.73159237649128]
  In current Semantic Communication systems, the construction of the knowledge base (KB) faces several issues. Here, we propose a LAM-based SC framework (LAM-SC) specifically designed for image data, where we first apply the segment anything model (SAM)-based KB (SKB) Then, we present an attention-based semantic integration (ASI) to weigh the semantic segments generated by SKB without human participation and integrate them as the semantic aware image.
  arXiv  Detail & Related papers  (2023-07-07T10:01:08Z)
• Few-shot Semantic Image Synthesis with Class Affinity Transfer [23.471210664024067]
  We propose a transfer method that leverages a model trained on a large source dataset to improve the learning ability on small target datasets. The class affinity matrix is introduced as a first layer to the source model to make it compatible with the target label maps. We apply our approach to GAN-based and diffusion-based architectures for semantic synthesis.
  arXiv  Detail & Related papers  (2023-04-05T09:24:45Z)
• Towards Semantic Communications: Deep Learning-Based Image Semantic Coding [42.453963827153856]
  We conceive the semantic communications for image data that is much more richer in semantics and bandwidth sensitive. We propose an reinforcement learning based adaptive semantic coding (RL-ASC) approach that encodes images beyond pixel level. Experimental results demonstrate that the proposed RL-ASC is noise robust and could reconstruct visually pleasant and semantic consistent image.
  arXiv  Detail & Related papers  (2022-08-08T12:29:55Z)
• Semantic Representation and Dependency Learning for Multi-Label Image Recognition [76.52120002993728]
  We propose a novel and effective semantic representation and dependency learning (SRDL) framework to learn category-specific semantic representation for each category. Specifically, we design a category-specific attentional regions (CAR) module to generate channel/spatial-wise attention matrices to guide model. We also design an object erasing (OE) module to implicitly learn semantic dependency among categories by erasing semantic-aware regions.
  arXiv  Detail & Related papers  (2022-04-08T00:55:15Z)
• A Unified Architecture of Semantic Segmentation and Hierarchical Generative Adversarial Networks for Expression Manipulation [52.911307452212256]
  We develop a unified architecture of semantic segmentation and hierarchical GANs. A unique advantage of our framework is that on forward pass the semantic segmentation network conditions the generative model. We evaluate our method on two challenging facial expression translation benchmarks, AffectNet and RaFD, and a semantic segmentation benchmark, CelebAMask-HQ.
  arXiv  Detail & Related papers  (2021-12-08T22:06:31Z)
• Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation [53.49821324597837]
  Weakly supervised semantic segmentation is a challenging problem that has been deeply studied in recent years. We present a Context Decoupling Augmentation ( CDA) method to change the inherent context in which the objects appear. To validate the effectiveness of the proposed method, extensive experiments on PASCAL VOC 2012 dataset with several alternative network architectures demonstrate that CDA can boost various popular WSSS methods to the new state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2021-03-02T15:05:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.