End-to-End Neuro-Symbolic Architecture for Image-to-Image Reasoning Tasks

• URL: http://arxiv.org/abs/2106.03121v1
• Date: Sun, 6 Jun 2021 13:27:33 GMT
• Title: End-to-End Neuro-Symbolic Architecture for Image-to-Image Reasoning Tasks
• Authors: Ananye Agarwal, Pradeep Shenoy, Mausam
• Abstract summary: We study neural-symbolic-neural models for reasoning tasks that require a conversion from an image input to an image output. We propose NSNnet, an architecture that combines an image reconstruction loss with a novel output encoder to generate a supervisory signal.
• Score: 15.649929244635269
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural models and symbolic algorithms have recently been combined for tasks requiring both perception and reasoning. Neural models ground perceptual input into a conceptual vocabulary, on which a classical reasoning algorithm is applied to generate output. A key limitation is that such neural-to-symbolic models can only be trained end-to-end for tasks where the output space is symbolic. In this paper, we study neural-symbolic-neural models for reasoning tasks that require a conversion from an image input (e.g., a partially filled sudoku) to an image output (e.g., the image of the completed sudoku). While designing such a three-step hybrid architecture may be straightforward, the key technical challenge is end-to-end training -- how to backpropagate without intermediate supervision through the symbolic component. We propose NSNnet, an architecture that combines an image reconstruction loss with a novel output encoder to generate a supervisory signal, develops update algorithms that leverage policy gradient methods for supervision, and optimizes loss using a novel subsampling heuristic. We experiment on problem settings where symbolic algorithms are easily specified: a visual maze solving task and a visual Sudoku solver where the supervision is in image form. Experiments show high accuracy with significantly less data compared to purely neural approaches.

Related papers

• Simple and Effective Transfer Learning for Neuro-Symbolic Integration [50.592338727912946]
  A potential solution to this issue is Neuro-Symbolic Integration (NeSy), where neural approaches are combined with symbolic reasoning. Most of these methods exploit a neural network to map perceptions to symbols and a logical reasoner to predict the output of the downstream task. They suffer from several issues, including slow convergence, learning difficulties with complex perception tasks, and convergence to local minima. This paper proposes a simple yet effective method to ameliorate these problems.
  arXiv  Detail & Related papers  (2024-02-21T15:51:01Z)
• The Role of Foundation Models in Neuro-Symbolic Learning and Reasoning [54.56905063752427]
  Neuro-Symbolic AI (NeSy) holds promise to ensure the safe deployment of AI systems. Existing pipelines that train the neural and symbolic components sequentially require extensive labelling. New architecture, NeSyGPT, fine-tunes a vision-language foundation model to extract symbolic features from raw data.
  arXiv  Detail & Related papers  (2024-02-02T20:33:14Z)
• Bridging Logic and Learning: A Neural-Symbolic Approach for Enhanced Reasoning in Neural Models (ASPER) [0.13053649021965597]
  This paper introduces an approach designed to improve the performance of neural models in learning reasoning tasks. It achieves this by integrating Answer Set Programming solvers and domain-specific expertise. The model shows a significant improvement in solving Sudoku puzzles using only 12 puzzles for training and testing.
  arXiv  Detail & Related papers  (2023-12-18T19:06:00Z)
• T-former: An Efficient Transformer for Image Inpainting [50.43302925662507]
  A class of attention-based network architectures, called transformer, has shown significant performance on natural language processing fields. In this paper, we design a novel attention linearly related to the resolution according to Taylor expansion, and based on this attention, a network called $T$-former is designed for image inpainting. Experiments on several benchmark datasets demonstrate that our proposed method achieves state-of-the-art accuracy while maintaining a relatively low number of parameters and computational complexity.
  arXiv  Detail & Related papers  (2023-05-12T04:10:42Z)
• Neural Knitworks: Patched Neural Implicit Representation Networks [1.0470286407954037]
  We propose Knitwork, an architecture for neural implicit representation learning of natural images that achieves image synthesis. To the best of our knowledge, this is the first implementation of a coordinate-based patch tailored for synthesis tasks such as image inpainting, super-resolution, and denoising. The results show that modeling natural images using patches, rather than pixels, produces results of higher fidelity.
  arXiv  Detail & Related papers  (2021-09-29T13:10:46Z)
• Fast Training of Neural Lumigraph Representations using Meta Learning [109.92233234681319]
  We develop a new neural rendering approach with the goal of quickly learning a high-quality representation which can also be rendered in real-time. Our approach, MetaNLR++, accomplishes this by using a unique combination of a neural shape representation and 2D CNN-based image feature extraction, aggregation, and re-projection. We show that MetaNLR++ achieves similar or better photorealistic novel view synthesis results in a fraction of the time that competing methods require.
  arXiv  Detail & Related papers  (2021-06-28T18:55:50Z)
• Neural Sparse Representation for Image Restoration [116.72107034624344]
  Inspired by the robustness and efficiency of sparse coding based image restoration models, we investigate the sparsity of neurons in deep networks. Our method structurally enforces sparsity constraints upon hidden neurons. Experiments show that sparse representation is crucial in deep neural networks for multiple image restoration tasks.
  arXiv  Detail & Related papers  (2020-06-08T05:15:17Z)
• Pyramid Attention Networks for Image Restoration [124.34970277136061]
  Self-similarity refers to the image prior widely used in image restoration algorithms. Recent advanced deep convolutional neural network based methods for image restoration do not take full advantage of self-similarities. We present a novel Pyramid Attention module for image restoration, which captures long-range feature correspondences from a multi-scale feature pyramid.
  arXiv  Detail & Related papers  (2020-04-28T21:12:36Z)
• Efficient Neural Architecture for Text-to-Image Synthesis [6.166295570030645]
  We show that an effective neural architecture can achieve state-of-the-art performance using a single stage training with a single generator and a single discriminator. Our work points a new direction for text-to-image research, which has not experimented with novel neural architectures recently.
  arXiv  Detail & Related papers  (2020-04-23T19:33:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.