GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and
reusing ModulEs
- URL: http://arxiv.org/abs/2311.04901v1
- Date: Wed, 8 Nov 2023 18:59:05 GMT
- Title: GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and
reusing ModulEs
- Authors: Zhenfang Chen, Rui Sun, Wenjun Liu, Yining Hong, Chuang Gan
- Abstract summary: We propose generative neuro-symbolic visual reasoning by growing and reusing modules.
The proposed model performs competitively on standard tasks like visual question answering and referring expression comprehension.
It is able to adapt to new visual reasoning tasks by observing a few training examples and reusing modules.
- Score: 64.49176353858792
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent works have shown that Large Language Models (LLMs) could empower
traditional neuro-symbolic models via programming capabilities to translate
language into module descriptions, thus achieving strong visual reasoning
results while maintaining the model's transparency and efficiency. However,
these models usually exhaustively generate the entire code snippet given each
new instance of a task, which is extremely ineffective. We propose generative
neuro-symbolic visual reasoning by growing and reusing modules. Specifically,
our model consists of three unique stages, module initialization, module
generation, and module execution. First, given a vision-language task, we adopt
LLMs to examine whether we could reuse and grow over established modules to
handle this new task. If not, we initialize a new module needed by the task and
specify the inputs and outputs of this new module. After that, the new module
is created by querying LLMs to generate corresponding code snippets that match
the requirements. In order to get a better sense of the new module's ability,
we treat few-shot training examples as test cases to see if our new module
could pass these cases. If yes, the new module is added to the module library
for future reuse. Finally, we evaluate the performance of our model on the
testing set by executing the parsed programs with the newly made visual modules
to get the results. We find the proposed model possesses several advantages.
First, it performs competitively on standard tasks like visual question
answering and referring expression comprehension; Second, the modules learned
from one task can be seamlessly transferred to new tasks; Last but not least,
it is able to adapt to new visual reasoning tasks by observing a few training
examples and reusing modules.
Related papers
- Configurable Foundation Models: Building LLMs from a Modular Perspective [115.63847606634268]
A growing tendency to decompose LLMs into numerous functional modules allows for inference with part of modules and dynamic assembly of modules to tackle complex tasks.
We coin the term brick to represent each functional module, designating the modularized structure as customizable foundation models.
We present four brick-oriented operations: retrieval and routing, merging, updating, and growing.
We find that the FFN layers follow modular patterns with functional specialization of neurons and functional neuron partitions.
arXiv Detail & Related papers (2024-09-04T17:01:02Z) - Unlocking Emergent Modularity in Large Language Models [27.12431620957652]
We show that standard Language Models (LMs) could be fine-tuned as their Mixture-of-Expert (MoEs) counterparts without introducing any extra parameters.
Our experiments demonstrate that fine-tuning EMoE effectively improves downstream in-domain and out-of-domain generalization compared with vanilla fine-tuning.
arXiv Detail & Related papers (2023-10-17T01:02:32Z) - CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules [51.82044734879657]
We propose CodeChain, a novel framework for inference that elicits modularized code generation through a chain of self-revisions.
We find that CodeChain can significantly boost both modularity as well as correctness of the generated solutions, achieving relative pass@1 improvements of 35% on APPS and 76% on CodeContests.
arXiv Detail & Related papers (2023-10-13T10:17:48Z) - ModuleFormer: Modularity Emerges from Mixture-of-Experts [60.6148988099284]
This paper proposes a new neural network architecture, ModuleFormer, to improve the efficiency and flexibility of large language models.
Unlike the previous SMoE-based modular language model, ModuleFormer can induce modularity from uncurated data.
arXiv Detail & Related papers (2023-06-07T17:59:57Z) - mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality [95.76661165594884]
mPLUG-Owl is a training paradigm that equips large language models (LLMs) with multi-modal abilities.
The training paradigm involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM.
Experimental results show that our model outperforms existing multi-modal models.
arXiv Detail & Related papers (2023-04-27T13:27:01Z) - Modular Deep Learning [120.36599591042908]
Transfer learning has recently become the dominant paradigm of machine learning.
It remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference.
Modular deep learning has emerged as a promising solution to these challenges.
arXiv Detail & Related papers (2023-02-22T18:11:25Z) - Neural Network Module Decomposition and Recomposition [35.21448933547118]
We propose a modularization method that decomposes a deep neural network (DNN) into small modules from a functionality perspective.
We demonstrate that the proposed method can decompose and recompose DNNs with high compression ratio and high accuracy.
arXiv Detail & Related papers (2021-12-25T08:36:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.