Improving Unsupervised Visual Program Inference with Code Rewriting Families

• URL: http://arxiv.org/abs/2309.14972v1
• Date: Tue, 26 Sep 2023 14:44:48 GMT
• Title: Improving Unsupervised Visual Program Inference with Code Rewriting Families
• Authors: Aditya Ganeshan, R. Kenny Jones and Daniel Ritchie
• Abstract summary: We show how code rewriting can be used to improve systems for inferring programs from visual data. We propose Sparse Intermittent Rewrite Injection (SIRI), a framework for unsupervised bootstrapped learning. We design a family of rewriters for visual programming domains: parameter optimization, code pruning, and code grafting.
• Score: 21.515789221802493
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Programs offer compactness and structure that makes them an attractive representation for visual data. We explore how code rewriting can be used to improve systems for inferring programs from visual data. We first propose Sparse Intermittent Rewrite Injection (SIRI), a framework for unsupervised bootstrapped learning. SIRI sparsely applies code rewrite operations over a dataset of training programs, injecting the improved programs back into the training set. We design a family of rewriters for visual programming domains: parameter optimization, code pruning, and code grafting. For three shape programming languages in 2D and 3D, we show that using SIRI with our family of rewriters improves performance: better reconstructions and faster convergence rates, compared with bootstrapped learning methods that do not use rewriters or use them naively. Finally, we demonstrate that our family of rewriters can be effectively used at test time to improve the output of SIRI predictions. For 2D and 3D CSG, we outperform or match the reconstruction performance of recent domain-specific neural architectures, while producing more parsimonious programs that use significantly fewer primitives.

Related papers

• CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay [12.499776923362461]
  We introduce a novel and scalable method for language model self-improvement called Code It (CodeIt) CodeIt iterates between 1) program sampling and hindsight relabeling, and 2) learning from prioritized experience replay. Applying CodeIt to the ARC dataset, we demonstrate that prioritized hindsight replay, along with pre-training and data-augmentation, leads to successful inter-task generalization.
  arXiv  Detail & Related papers  (2024-02-07T13:55:27Z)
• ReGAL: Refactoring Programs to Discover Generalizable Abstractions [59.05769810380928]
  Generalizable Abstraction Learning (ReGAL) is a method for learning a library of reusable functions via codeization. We find that the shared function libraries discovered by ReGAL make programs easier to predict across diverse domains. For CodeLlama-13B, ReGAL results in absolute accuracy increases of 11.5% on LOGO, 26.1% on date understanding, and 8.1% on TextCraft, outperforming GPT-3.5 in two of three domains.
  arXiv  Detail & Related papers  (2024-01-29T18:45:30Z)
• Continual Referring Expression Comprehension via Dual Modular Memorization [133.46886428655426]
  Referring Expression (REC) aims to localize an image region of a given object described by a natural-language expression. Existing REC algorithms make a strong assumption that training data feeding into a model are given upfront, which degrades its practicality for real-world scenarios. In this paper, we propose Continual Referring Expression (CREC), a new setting for REC, where a model is learning on a stream of incoming tasks. In order to continuously improve the model on sequential tasks without forgetting prior learned knowledge and without repeatedly re-training from a scratch, we propose an effective baseline method named Dual Modular Memorization
  arXiv  Detail & Related papers  (2023-11-25T02:58:51Z)
• Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation [23.31928097405939]
  We use a language-model-infused scaffolding program to improve itself. A variety of self-improvement strategies are proposed by the language model. It demonstrates that a modern language model, GPT-4, is capable of writing code that can call itself to improve itself.
  arXiv  Detail & Related papers  (2023-10-03T17:59:32Z)
• Retrieval-Enhanced Contrastive Vision-Text Models [61.783728119255365]
  We propose to equip vision-text models with the ability to refine their embedding with cross-modal retrieved information from a memory at inference time. Remarkably, we show that this can be done with a light-weight, single-layer, fusion transformer on top of a frozen CLIP. Our experiments validate that our retrieval-enhanced contrastive (RECO) training improves CLIP performance substantially on several challenging fine-grained tasks.
  arXiv  Detail & Related papers  (2023-06-12T15:52:02Z)
• PERFOGRAPH: A Numerical Aware Program Graph Representation for Performance Optimization and Program Analysis [12.778336318809092]
  A key challenge in adopting the latest machine learning methods is the representation of programming languages. To overcome the limitations and challenges of current program representations, we propose a graph-based program representation called PERFOGRAPH. PerFOGRAPH can capture numerical information and the aggregate data structure by introducing new nodes and edges.
  arXiv  Detail & Related papers  (2023-05-31T21:59:50Z)
• The Wisdom of Hindsight Makes Language Models Better Instruction Followers [84.9120606803906]
  Reinforcement learning has seen wide success in finetuning large language models to better align with instructions via human feedback. In this paper, we consider an alternative approach: converting feedback to instruction by relabeling the original one and training the model for better alignment in a supervised manner. We propose Hindsight Instruction Relabeling (HIR), a novel algorithm for aligning language models with instructions.
  arXiv  Detail & Related papers  (2023-02-10T12:16:38Z)
• SiRi: A Simple Selective Retraining Mechanism for Transformer-based Visual Grounding [131.0977050185209]
  Selective Retraining (SiRi) can significantly outperform previous approaches on three popular benchmarks. SiRi performs surprisingly superior even with limited training data. We also extend it to transformer-based visual grounding models and other vision-language tasks to verify the validity.
  arXiv  Detail & Related papers  (2022-07-27T07:01:01Z)
• Circa: Stochastic ReLUs for Private Deep Learning [6.538025863698682]
  We re-think the ReLU computation and propose optimizations for PI tailored to neural networks. Specifically, we reformulate ReLU as an approximate sign test and introduce a novel truncation method for the sign test. We demonstrate improvements of up to 4.7x storage and 3x runtime over baseline implementations.
  arXiv  Detail & Related papers  (2021-06-15T22:52:45Z)
• Learning to learn generative programs with Memoised Wake-Sleep [52.439550543743536]
  We study a class of neuro-symbolic generative models in which neural networks are used both for inference and as priors over symbolic, data-generating programs. We propose the Memoised Wake-Sleep (MWS) algorithm, which extends Wake Sleep by explicitly storing and reusing the best programs discovered by the inference network throughout training.
  arXiv  Detail & Related papers  (2020-07-06T23:51:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.