Hiding Data Helps: On the Benefits of Masking for Sparse Coding

• URL: http://arxiv.org/abs/2302.12715v2
• Date: Thu, 1 Jun 2023 14:45:19 GMT
• Title: Hiding Data Helps: On the Benefits of Masking for Sparse Coding
• Authors: Muthu Chidambaram, Chenwei Wu, Yu Cheng, Rong Ge
• Abstract summary: We show that in the presence of noise, minimizing the standard dictionary learning objective can fail to recover the elements of the ground-truth dictionary in the over-realized regime. We propose a novel masking objective for which recovering the ground-truth dictionary is in fact optimal as the signal increases for a large class of data-generating processes.
• Score: 22.712098918769243
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse coding, which refers to modeling a signal as sparse linear combinations of the elements of a learned dictionary, has proven to be a successful (and interpretable) approach in applications such as signal processing, computer vision, and medical imaging. While this success has spurred much work on provable guarantees for dictionary recovery when the learned dictionary is the same size as the ground-truth dictionary, work on the setting where the learned dictionary is larger (or over-realized) with respect to the ground truth is comparatively nascent. Existing theoretical results in this setting have been constrained to the case of noise-less data. We show in this work that, in the presence of noise, minimizing the standard dictionary learning objective can fail to recover the elements of the ground-truth dictionary in the over-realized regime, regardless of the magnitude of the signal in the data-generating process. Furthermore, drawing from the growing body of work on self-supervised learning, we propose a novel masking objective for which recovering the ground-truth dictionary is in fact optimal as the signal increases for a large class of data-generating processes. We corroborate our theoretical results with experiments across several parameter regimes showing that our proposed objective also enjoys better empirical performance than the standard reconstruction objective.

Related papers

• Lightweight Conceptual Dictionary Learning for Text Classification Using Information Compression [15.460141768587663]
  We propose a lightweight supervised dictionary learning framework for text classification based on data compression and representation. We evaluate our algorithm's information-theoretic performance using information bottleneck principles and introduce the information plane area rank (IPAR) as a novel metric to quantify the information-theoretic performance.
  arXiv  Detail & Related papers  (2024-04-28T10:11:52Z)
• An Analysis of BPE Vocabulary Trimming in Neural Machine Translation [56.383793805299234]
  vocabulary trimming is a postprocessing step that replaces rare subwords with their component subwords. We show that vocabulary trimming fails to improve performance and is even prone to incurring heavy degradation.
  arXiv  Detail & Related papers  (2024-03-30T15:29:49Z)
• Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT [59.245414547751636]
  We propose a circuit discovery framework alternative to activation patching. Our framework suffers less from out-of-distribution and proves to be more efficient in terms of complexity. We dig in a small transformer trained on a synthetic task named Othello and find a number of human-understandable fine-grained circuits inside of it.
  arXiv  Detail & Related papers  (2024-02-19T15:04:53Z)
• Simple Alternating Minimization Provably Solves Complete Dictionary Learning [13.056764072568749]
  This paper focuses on complete dictionary problem, where the goal is to reparametrize a set of given signals as linear combinations of atoms from a learned dictionary. There are two main challenges faced by theoretical and practical dictionary learning: the lack of theoretical guarantees for practically-used algorithms, and poor scalability when dealing with huge-scale datasets.
  arXiv  Detail & Related papers  (2022-10-23T18:30:45Z)
• DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection [118.36746273425354]
  This paper presents a paralleled visual-concept pre-training method for open-world detection by resorting to knowledge enrichment from a designed concept dictionary. By enriching the concepts with their descriptions, we explicitly build the relationships among various concepts to facilitate the open-domain learning. The proposed framework demonstrates strong zero-shot detection performances, e.g., on the LVIS dataset, our DetCLIP-T outperforms GLIP-T by 9.9% mAP and obtains a 13.5% improvement on rare categories.
  arXiv  Detail & Related papers  (2022-09-20T02:01:01Z)
• Discriminative Dictionary Learning based on Statistical Methods [0.0]
  Sparse Representation (SR) of signals or data has a well founded theory with rigorous mathematical error bounds and proofs. Training dictionaries such that they represent each class of signals with minimal loss is called Dictionary Learning (DL) MOD and K-SVD have been successfully used in reconstruction based applications in image processing like image "denoising", "inpainting"
  arXiv  Detail & Related papers  (2021-11-17T10:45:10Z)
• Deep learning based dictionary learning and tomographic image reconstruction [0.0]
  This work presents an approach for image reconstruction in clinical low-dose tomography that combines principles from sparse signal processing with ideas from deep learning. First, we describe sparse signal representation in terms of dictionaries from a statistical perspective and interpret dictionary learning as a process of aligning distribution that arises from a generative model with empirical distribution of true signals. As a result we can see that sparse coding with learned dictionaries resembles a specific variational autoencoder, where the decoder is a linear function and the encoder is a sparse coding algorithm.
  arXiv  Detail & Related papers  (2021-08-26T12:10:17Z)
• PUDLE: Implicit Acceleration of Dictionary Learning by Backpropagation [4.081440927534577]
  This paper offers the first theoretical proof for empirical results through PUDLE, a Provable Unfolded Dictionary LEarning method. We highlight the minimization impact of loss, unfolding, and backpropagation on convergence. We complement our findings through synthetic and image denoising experiments.
  arXiv  Detail & Related papers  (2021-05-31T18:49:58Z)
• When Dictionary Learning Meets Deep Learning: Deep Dictionary Learning and Coding Network for Image Recognition with Limited Data [74.75557280245643]
  We present a new Deep Dictionary Learning and Coding Network (DDLCN) for image recognition tasks with limited data. We empirically compare DDLCN with several leading dictionary learning methods and deep learning models. Experimental results on five popular datasets show that DDLCN achieves competitive results compared with state-of-the-art methods when the training data is limited.
  arXiv  Detail & Related papers  (2020-05-21T23:12:10Z)
• Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressions [87.33156149634392]
  We critically examine RefCOg, a standard benchmark for visual referring expression recognition. We show that 83.7% of test instances do not require reasoning on linguistic structure. We propose two methods, one based on contrastive learning and the other based on multi-task learning, to increase the robustness of ViLBERT.
  arXiv  Detail & Related papers  (2020-05-04T17:09:15Z)
• Lexical Sememe Prediction using Dictionary Definitions by Capturing Local Semantic Correspondence [94.79912471702782]
  Sememes, defined as the minimum semantic units of human languages, have been proven useful in many NLP tasks. We propose a Sememe Correspondence Pooling (SCorP) model, which is able to capture this kind of matching to predict sememes. We evaluate our model and baseline methods on a famous sememe KB HowNet and find that our model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-01-16T17:30:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.