Learning Signal-Agnostic Manifolds of Neural Fields

• URL: http://arxiv.org/abs/2111.06387v1
• Date: Thu, 11 Nov 2021 18:57:40 GMT
• Title: Learning Signal-Agnostic Manifolds of Neural Fields
• Authors: Yilun Du, Katherine M. Collins, Joshua B. Tenenbaum, Vincent Sitzmann
• Abstract summary: We leverage neural fields to capture the underlying structure in image, shape, audio and cross-modal audiovisual domains. We show that by walking across the underlying manifold of GEM, we may generate new samples in our signal domains.
• Score: 50.066449953522685
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks have been used widely to learn the latent structure of datasets, across modalities such as images, shapes, and audio signals. However, existing models are generally modality-dependent, requiring custom architectures and objectives to process different classes of signals. We leverage neural fields to capture the underlying structure in image, shape, audio and cross-modal audiovisual domains in a modality-independent manner. We cast our task as one of learning a manifold, where we aim to infer a low-dimensional, locally linear subspace in which our data resides. By enforcing coverage of the manifold, local linearity, and local isometry, our model -- dubbed GEM -- learns to capture the underlying structure of datasets across modalities. We can then travel along linear regions of our manifold to obtain perceptually consistent interpolations between samples, and can further use GEM to recover points on our manifold and glean not only diverse completions of input images, but cross-modal hallucinations of audio or image signals. Finally, we show that by walking across the underlying manifold of GEM, we may generate new samples in our signal domains. Code and additional results are available at https://yilundu.github.io/gem/.

Related papers

• nnMamba: 3D Biomedical Image Segmentation, Classification and Landmark Detection with State Space Model [24.955052600683423]
  In this paper, we introduce nnMamba, a novel architecture that integrates the strengths of CNNs and the advanced long-range modeling capabilities of State Space Sequence Models (SSMs) Experiments on 6 datasets demonstrate nnMamba's superiority over state-of-the-art methods in a suite of challenging tasks, including 3D image segmentation, classification, and landmark detection.
  arXiv  Detail & Related papers  (2024-02-05T21:28:47Z)
• Keypoint-Augmented Self-Supervised Learning for Medical Image Segmentation with Limited Annotation [21.203307064937142]
  We present a keypointaugmented fusion layer that extracts representations preserving both short- and long-range self-attention. In particular, we augment the CNN feature map at multiple scales by incorporating an additional input that learns long-range spatial selfattention. Our method further outperforms existing SSL methods by producing more robust self-attention.
  arXiv  Detail & Related papers  (2023-10-02T22:31:30Z)
• Extracting Semantic Knowledge from GANs with Unsupervised Learning [65.32631025780631]
  Generative Adversarial Networks (GANs) encode semantics in feature maps in a linearly separable form. We propose a novel clustering algorithm, named KLiSH, which leverages the linear separability to cluster GAN's features. KLiSH succeeds in extracting fine-grained semantics of GANs trained on datasets of various objects.
  arXiv  Detail & Related papers  (2022-11-30T03:18:16Z)
• Contrastive Neighborhood Alignment [81.65103777329874]
  We present Contrastive Neighborhood Alignment (CNA), a manifold learning approach to maintain the topology of learned features. The target model aims to mimic the local structure of the source representation space using a contrastive loss. CNA is illustrated in three scenarios: manifold learning, where the model maintains the local topology of the original data in a dimension-reduced space; model distillation, where a small student model is trained to mimic a larger teacher; and legacy model update, where an older model is replaced by a more powerful one.
  arXiv  Detail & Related papers  (2022-01-06T04:58:31Z)
• Self-supervised Audiovisual Representation Learning for Remote Sensing Data [96.23611272637943]
  We propose a self-supervised approach for pre-training deep neural networks in remote sensing. By exploiting the correspondence between geo-tagged audio recordings and remote sensing, this is done in a completely label-free manner. We show that our approach outperforms existing pre-training strategies for remote sensing imagery.
  arXiv  Detail & Related papers  (2021-08-02T07:50:50Z)
• Generalized Zero-Shot Learning using Multimodal Variational Auto-Encoder with Semantic Concepts [0.9054540533394924]
  Recent techniques try to learn a cross-modal mapping between the semantic space and the image space. We propose a Multimodal Variational Auto-Encoder (M-VAE) which can learn the shared latent space of image features and the semantic space. Our results show that our proposed model outperforms the current state-of-the-art approaches for generalized zero-shot learning.
  arXiv  Detail & Related papers  (2021-06-26T20:08:37Z)
• PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning [109.84770951839289]
  We present PredRNN, a new recurrent network for learning visual dynamics from historical context. We show that our approach obtains highly competitive results on three standard datasets.
  arXiv  Detail & Related papers  (2021-03-17T08:28:30Z)
• PGL: Prior-Guided Local Self-supervised Learning for 3D Medical Image Segmentation [87.50205728818601]
  We propose a PriorGuided Local (PGL) self-supervised model that learns the region-wise local consistency in the latent feature space. Our PGL model learns the distinctive representations of local regions, and hence is able to retain structural information.
  arXiv  Detail & Related papers  (2020-11-25T11:03:11Z)
• Deep Learning based Segmentation of Fish in Noisy Forward Looking MBES Images [1.5469452301122177]
  We build on recent advances in Deep Learning (DL) and Convolutional Neural Networks (CNNs) for semantic segmentation. We demonstrate an end-to-end approach for a fish/non-fish probability prediction for all range-azimuth positions projected by an imaging sonar. We show that our model proves the desired performance and has learned to harness the importance of semantic context.
  arXiv  Detail & Related papers  (2020-06-16T09:57:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.