An Indirect Rate-Distortion Characterization for Semantic Sources: General Model and the Case of Gaussian Observation

• URL: http://arxiv.org/abs/2201.12477v1
• Date: Sat, 29 Jan 2022 02:14:24 GMT
• Title: An Indirect Rate-Distortion Characterization for Semantic Sources: General Model and the Case of Gaussian Observation
• Authors: Jiakun Liu, Shuo Shao, Wenyi Zhang, H. Vincent Poor
• Abstract summary: Source model is motivated by the recent surge of interest in the semantic aspect of information. intrinsic state corresponds to the semantic feature of the source, which in general is not observable. Rate-distortion function is the semantic rate-distortion function of the source.
• Score: 83.93224401261068
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A new source model, which consists of an intrinsic state part and an extrinsic observation part, is proposed and its information-theoretic characterization, namely its rate-distortion function, is defined and analyzed. Such a source model is motivated by the recent surge of interest in the semantic aspect of information: the intrinsic state corresponds to the semantic feature of the source, which in general is not observable but can only be inferred from the extrinsic observation. There are two distortion measures, one between the intrinsic state and its reproduction, and the other between the extrinsic observation and its reproduction. Under a given code rate, the tradeoff between these two distortion measures is characterized by the rate-distortion function, which is solved via the indirect rate-distortion theory and is termed as the semantic rate-distortion function of the source. As an application of the general model and its analysis, the case of Gaussian extrinsic observation is studied, assuming a linear relationship between the intrinsic state and the extrinsic observation, under a quadratic distortion structure. The semantic rate-distortion function is shown to be the solution of a convex programming programming with respect to an error covariance matrix, and a reverse water-filling type of solution is provided when the model further satisfies a diagonalizability condition.

Related papers

• Geometric Insights into Focal Loss: Reducing Curvature for Enhanced Model Calibration [1.642094639107215]
  The confidence level of a model in a classification problem is often given by the output vector of a softmax function for convenience. This problem is called model calibration and has been studied extensively. We show that focal loss reduces the curvature of the loss surface in training the model.
  arXiv  Detail & Related papers  (2024-05-01T10:53:54Z)
• Demystifying Disagreement-on-the-Line in High Dimensions [34.103373453782744]
  We develop a theoretical foundation for analyzing disagreement in high-dimensional random features regression. Experiments on CIFAR-10-C, Tiny ImageNet-C, and Camelyon17 are consistent with our theory and support the universality of the theoretical findings.
  arXiv  Detail & Related papers  (2023-01-31T02:31:18Z)
• On the Strong Correlation Between Model Invariance and Generalization [54.812786542023325]
  Generalization captures a model's ability to classify unseen data. Invariance measures consistency of model predictions on transformations of the data. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets.
  arXiv  Detail & Related papers  (2022-07-14T17:08:25Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• Closed-form discovery of structural errors in models of chaotic systems by integrating Bayesian sparse regression and data assimilation [0.0]
  We introduce a framework named MEDIDA: Model Error Discovery with Interpretability and Data Assimilation. In MEDIDA, first the model error is estimated from differences between the observed states and model-predicted states. If observations are noisy, a data assimilation technique such as ensemble Kalman filter (EnKF) is first used to provide a noise-free analysis state of the system. Finally, an equation-discovery technique, such as the relevance vector machine (RVM), a sparsity-promoting Bayesian method, is used to identify an interpretable, parsimonious, closed
  arXiv  Detail & Related papers  (2021-10-01T17:19:28Z)
• The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks [51.1848572349154]
  neural network models that perfectly fit noisy data can generalize well to unseen test data. We consider interpolating two-layer linear neural networks trained with gradient flow on the squared loss and derive bounds on the excess risk.
  arXiv  Detail & Related papers  (2021-08-25T22:01:01Z)
• On the Role of Optimization in Double Descent: A Least Squares Study [30.44215064390409]
  We show an excess risk bound for the descent gradient solution of the least squares objective. We find that in case of noiseless regression, double descent is explained solely by optimization-related quantities. We empirically explore if our predictions hold for neural networks.
  arXiv  Detail & Related papers  (2021-07-27T09:13:11Z)
• Leveraging Global Parameters for Flow-based Neural Posterior Estimation [90.21090932619695]
  Inferring the parameters of a model based on experimental observations is central to the scientific method. A particularly challenging setting is when the model is strongly indeterminate, i.e., when distinct sets of parameters yield identical observations. We present a method for cracking such indeterminacy by exploiting additional information conveyed by an auxiliary set of observations sharing global parameters.
  arXiv  Detail & Related papers  (2021-02-12T12:23:13Z)
• Semiparametric Nonlinear Bipartite Graph Representation Learning with Provable Guarantees [106.91654068632882]
  We consider the bipartite graph and formalize its representation learning problem as a statistical estimation problem of parameters in a semiparametric exponential family distribution. We show that the proposed objective is strongly convex in a neighborhood around the ground truth, so that a gradient descent-based method achieves linear convergence rate. Our estimator is robust to any model misspecification within the exponential family, which is validated in extensive experiments.
  arXiv  Detail & Related papers  (2020-03-02T16:40:36Z)
• Generalisation error in learning with random features and the hidden manifold model [23.71637173968353]
  We study generalised linear regression and classification for a synthetically generated dataset. We consider the high-dimensional regime and using the replica method from statistical physics. We show how to obtain the so-called double descent behaviour for logistic regression with a peak at the threshold. We discuss the role played by correlations in the data generated by the hidden manifold model.
  arXiv  Detail & Related papers  (2020-02-21T14:49:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.