Tackling Distribution Shifts in Task-Oriented Communication with Information Bottleneck

• URL: http://arxiv.org/abs/2405.09514v1
• Date: Wed, 15 May 2024 17:07:55 GMT
• Title: Tackling Distribution Shifts in Task-Oriented Communication with Information Bottleneck
• Authors: Hongru Li, Jiawei Shao, Hengtao He, Shenghui Song, Jun Zhang, Khaled B. Letaief,
• Abstract summary: We propose a novel approach based on the information bottleneck (IB) principle and invariant risk minimization (IRM) framework. The proposed method aims to extract compact and informative features that possess high capability for effective domain-shift generalization. We show that the proposed scheme outperforms state-of-the-art approaches and achieves a better rate-distortion tradeoff.
• Score: 28.661084093544684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Task-oriented communication aims to extract and transmit task-relevant information to significantly reduce the communication overhead and transmission latency. However, the unpredictable distribution shifts between training and test data, including domain shift and semantic shift, can dramatically undermine the system performance. In order to tackle these challenges, it is crucial to ensure that the encoded features can generalize to domain-shifted data and detect semanticshifted data, while remaining compact for transmission. In this paper, we propose a novel approach based on the information bottleneck (IB) principle and invariant risk minimization (IRM) framework. The proposed method aims to extract compact and informative features that possess high capability for effective domain-shift generalization and accurate semantic-shift detection without any knowledge of the test data during training. Specifically, we propose an invariant feature encoding approach based on the IB principle and IRM framework for domainshift generalization, which aims to find the causal relationship between the input data and task result by minimizing the complexity and domain dependence of the encoded feature. Furthermore, we enhance the task-oriented communication with the label-dependent feature encoding approach for semanticshift detection which achieves joint gains in IB optimization and detection performance. To avoid the intractable computation of the IB-based objective, we leverage variational approximation to derive a tractable upper bound for optimization. Extensive simulation results on image classification tasks demonstrate that the proposed scheme outperforms state-of-the-art approaches and achieves a better rate-distortion tradeoff.

Related papers

• Distributed Event-Based Learning via ADMM [11.461617927469316]
  We consider a distributed learning problem, where agents minimize a global objective function by exchanging information over a network. Our approach has two distinct features: (i) It substantially reduces communication by triggering communication only when necessary, and (ii) it is agnostic to the data-distribution among the different agents.
  arXiv  Detail & Related papers  (2024-05-17T08:30:28Z)
• PREM: A Simple Yet Effective Approach for Node-Level Graph Anomaly Detection [65.24854366973794]
  Node-level graph anomaly detection (GAD) plays a critical role in identifying anomalous nodes from graph-structured data in domains such as medicine, social networks, and e-commerce. We introduce a simple method termed PREprocessing and Matching (PREM for short) to improve the efficiency of GAD. Our approach streamlines GAD, reducing time and memory consumption while maintaining powerful anomaly detection capabilities.
  arXiv  Detail & Related papers  (2023-10-18T02:59:57Z)
• Analysis and Optimization of Wireless Federated Learning with Data Heterogeneity [72.85248553787538]
  This paper focuses on performance analysis and optimization for wireless FL, considering data heterogeneity, combined with wireless resource allocation. We formulate the loss function minimization problem, under constraints on long-term energy consumption and latency, and jointly optimize client scheduling, resource allocation, and the number of local training epochs (CRE) Experiments on real-world datasets demonstrate that the proposed algorithm outperforms other benchmarks in terms of the learning accuracy and energy consumption.
  arXiv  Detail & Related papers  (2023-08-04T04:18:01Z)
• Compressed Regression over Adaptive Networks [58.79251288443156]
  We derive the performance achievable by a network of distributed agents that solve, adaptively and in the presence of communication constraints, a regression problem. We devise an optimized allocation strategy where the parameters necessary for the optimization can be learned online by the agents.
  arXiv  Detail & Related papers  (2023-04-07T13:41:08Z)
• Joint Task and Data Oriented Semantic Communications: A Deep Separate Source-channel Coding Scheme [17.4244108919728]
  To serve both the data transmission and semantic tasks, joint data compression and semantic analysis has become pivotal issue in semantic communications. This paper proposes a deep separate source-channel coding framework for the joint task and data oriented semantic communications. An iterative training algorithm is proposed to tackle the overfitting issue of deep learning models.
  arXiv  Detail & Related papers  (2023-02-27T08:34:37Z)
• Magnitude Matters: Fixing SIGNSGD Through Magnitude-Aware Sparsification in the Presence of Data Heterogeneity [60.791736094073]
  Communication overhead has become one of the major bottlenecks in the distributed training of deep neural networks. We propose a magnitude-driven sparsification scheme, which addresses the non-convergence issue of SIGNSGD. The proposed scheme is validated through experiments on Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets.
  arXiv  Detail & Related papers  (2023-02-19T17:42:35Z)
• Balancing Discriminability and Transferability for Source-Free Domain Adaptation [55.143687986324935]
  Conventional domain adaptation (DA) techniques aim to improve domain transferability by learning domain-invariant representations. The requirement of simultaneous access to labeled source and unlabeled target renders them unsuitable for the challenging source-free DA setting. We derive novel insights to show that a mixup between original and corresponding translated generic samples enhances the discriminability-transferability trade-off.
  arXiv  Detail & Related papers  (2022-06-16T09:06:22Z)
• Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
  Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains. We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
  arXiv  Detail & Related papers  (2021-12-12T06:11:16Z)
• Task-Oriented Communication for Multi-Device Cooperative Edge Inference [14.249444124834719]
  cooperative edge inference can overcome the limited sensing capability of a single device, but it substantially increases the communication overhead and may incur excessive latency. We propose a learning-based communication scheme that optimize local feature extraction and distributed feature encoding in a task-oriented manner.
  arXiv  Detail & Related papers  (2021-09-01T03:56:20Z)
• Robust Representation Learning via Perceptual Similarity Metrics [18.842322467828502]
  Contrastive Input Morphing (CIM) is a representation learning framework that learns input-space transformations of the data. We show that CIM is complementary to other mutual information-based representation learning techniques.
  arXiv  Detail & Related papers  (2021-06-11T21:45:44Z)
• Learning Task-Oriented Communication for Edge Inference: An Information Bottleneck Approach [3.983055670167878]
  A low-end edge device transmits the extracted feature vector of a local data sample to a powerful edge server for processing. It is critical to encode the data into an informative and compact representation for low-latency inference given the limited bandwidth. We propose a learning-based communication scheme that jointly optimize feature extraction, source coding, and channel coding.
  arXiv  Detail & Related papers  (2021-02-08T12:53:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.