Explaining black box text modules in natural language with language models

• URL: http://arxiv.org/abs/2305.09863v2
• Date: Wed, 15 Nov 2023 17:19:10 GMT
• Title: Explaining black box text modules in natural language with language models
• Authors: Chandan Singh, Aliyah R. Hsu, Richard Antonello, Shailee Jain, Alexander G. Huth, Bin Yu, Jianfeng Gao
• Abstract summary: "Black box" indicates that we only have access to the module's inputs/outputs. "SASC" is a method that takes in a text module and returns a natural language explanation of the module's selectivity along with a score for how reliable the explanation is. We show that SASC can generate explanations for the response of individual fMRI voxels to language stimuli, with potential applications to fine-grained brain mapping.
• Score: 86.14329261605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have demonstrated remarkable prediction performance for a growing array of tasks. However, their rapid proliferation and increasing opaqueness have created a growing need for interpretability. Here, we ask whether we can automatically obtain natural language explanations for black box text modules. A "text module" is any function that maps text to a scalar continuous value, such as a submodule within an LLM or a fitted model of a brain region. "Black box" indicates that we only have access to the module's inputs/outputs. We introduce Summarize and Score (SASC), a method that takes in a text module and returns a natural language explanation of the module's selectivity along with a score for how reliable the explanation is. We study SASC in 3 contexts. First, we evaluate SASC on synthetic modules and find that it often recovers ground truth explanations. Second, we use SASC to explain modules found within a pre-trained BERT model, enabling inspection of the model's internals. Finally, we show that SASC can generate explanations for the response of individual fMRI voxels to language stimuli, with potential applications to fine-grained brain mapping. All code for using SASC and reproducing results is made available on Github.

Related papers

• GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and reusing ModulEs [64.49176353858792]
  We propose generative neuro-symbolic visual reasoning by growing and reusing modules. The proposed model performs competitively on standard tasks like visual question answering and referring expression comprehension. It is able to adapt to new visual reasoning tasks by observing a few training examples and reusing modules.
  arXiv  Detail & Related papers  (2023-11-08T18:59:05Z)
• Augmented Language Models: a Survey [55.965967655575454]
  This survey reviews works in which language models (LMs) are augmented with reasoning skills and the ability to use tools. We refer to them as Augmented Language Models (ALMs) The missing token objective allows ALMs to learn to reason, use tools, and even act, while still performing standard natural language tasks.
  arXiv  Detail & Related papers  (2023-02-15T18:25:52Z)
• Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning [80.59607794927363]
  We propose a novel image captioner: learning to Collocate Visual-Linguistic Neural Modules (LNCVM) Unlike the rewidely used neural module networks in VQA, the task of collocating visual-linguistic modules is more challenging. Our CVLNM is more effective,. achieving a new state-of-the-art 129.5 CIDEr-D, and more robust. Experiments on the MS-COCO dataset show that our CVLNM is more effective,. achieving a new state-of-the-art 129.5 CIDEr
  arXiv  Detail & Related papers  (2022-10-04T03:09:50Z)
• Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models [61.480085460269514]
  We propose a framework for building interpretable systems that learn to solve complex tasks by decomposing them into simpler ones solvable by existing models. We use this framework to build ModularQA, a system that can answer multi-hop reasoning questions by decomposing them into sub-questions answerable by a neural factoid single-span QA model and a symbolic calculator.
  arXiv  Detail & Related papers  (2020-09-01T23:45:42Z)
• Learning to Discretely Compose Reasoning Module Networks for Video Captioning [81.81394228898591]
  We propose a novel visual reasoning approach for video captioning, named Reasoning Module Networks (RMN) RMN employs three sophisticated RM-temporal reasoning, and 2) a dynamic and discrete module selector trained by a linguistic loss with a Gumbel approximation.
  arXiv  Detail & Related papers  (2020-07-17T15:27:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.