On the Information Content of Predictions in Word Analogy Tests
- URL: http://arxiv.org/abs/2210.09972v1
- Date: Tue, 18 Oct 2022 16:32:25 GMT
- Title: On the Information Content of Predictions in Word Analogy Tests
- Authors: Jugurta Montalv\~ao
- Abstract summary: An approach is proposed to quantify, in bits of information, the actual relevance of analogies in analogy tests.
The main component of this approach is a softaccuracy estimator that also yields entropy estimates with compensated biases.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: An approach is proposed to quantify, in bits of information, the actual
relevance of analogies in analogy tests. The main component of this approach is
a softaccuracy estimator that also yields entropy estimates with compensated
biases. Experimental results obtained with pre-trained GloVe 300-D vectors and
two public analogy test sets show that proximity hints are much more relevant
than analogies in analogy tests, from an information content perspective.
Accordingly, a simple word embedding model is used to predict that analogies
carry about one bit of information, which is experimentally corroborated.
Related papers
- Diffusion posterior sampling for simulation-based inference in tall data settings [53.17563688225137]
Simulation-based inference ( SBI) is capable of approximating the posterior distribution that relates input parameters to a given observation.
In this work, we consider a tall data extension in which multiple observations are available to better infer the parameters of the model.
We compare our method to recently proposed competing approaches on various numerical experiments and demonstrate its superiority in terms of numerical stability and computational cost.
arXiv Detail & Related papers (2024-04-11T09:23:36Z) - StoryAnalogy: Deriving Story-level Analogies from Large Language Models
to Unlock Analogical Understanding [72.38872974837462]
We evaluate the ability to identify and generate analogies by constructing a first-of-its-kind large-scale story-level analogy corpus.
textscStory Analogy contains 24K story pairs from diverse domains with human annotations on two similarities from the extended Structure-Mapping Theory.
We observe that the data in textscStory Analogy can improve the quality of analogy generation in large language models.
arXiv Detail & Related papers (2023-10-19T16:29:23Z) - Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance [0.0]
We test several ways to learn basic analogical reasoning, specifically focusing on analogies that are more typical of what is used to evaluate analogical reasoning in humans.
Our experiments find that models are able to learn analogical reasoning, even with a small amount of data.
arXiv Detail & Related papers (2023-10-09T10:34:38Z) - ANALOGYKB: Unlocking Analogical Reasoning of Language Models with A Million-scale Knowledge Base [51.777618249271725]
ANALOGYKB is a million-scale analogy knowledge base derived from existing knowledge graphs (KGs)
It identifies two types of analogies from the KGs: 1) analogies of the same relations, which can be directly extracted from the KGs, and 2) analogies of analogous relations, which are identified with a selection and filtering pipeline enabled by large language models (LLMs)
arXiv Detail & Related papers (2023-05-10T09:03:01Z) - Scientific and Creative Analogies in Pretrained Language Models [24.86477727507679]
This paper examines the encoding of analogy in large-scale pretrained language models, such as BERT and GPT-2.
We introduce the Scientific and Creative Analogy dataset (SCAN), a novel analogy dataset containing systematic mappings of multiple attributes and relational structures across dissimilar domains.
We find that state-of-the-art LMs achieve low performance on these complex analogy tasks, highlighting the challenges still posed by analogy understanding.
arXiv Detail & Related papers (2022-11-28T12:49:44Z) - Analogies and Feature Attributions for Model Agnostic Explanation of
Similarity Learners [29.63747822793279]
We propose a method that provides feature attributions to explain the similarity between a pair of inputs as determined by a black box similarity learner.
Here the goal is to identify diverse analogous pairs of examples that share the same level of similarity as the input pair.
We prove that our analogy objective function is submodular, making the search for good-quality analogies efficient.
arXiv Detail & Related papers (2022-02-02T17:28:56Z) - Probability distributions for analog-to-target distances [0.12314765641075436]
We show that dimensionality plays a role on the size of the catalog needed to find good analogs.
We show that dimensionality plays a role on the size of the catalog needed to find good analogs, and also on the relative means and variances of the $K$-best analogs.
arXiv Detail & Related papers (2021-01-26T09:10:12Z) - Toward Scalable and Unified Example-based Explanation and Outlier
Detection [128.23117182137418]
We argue for a broader adoption of prototype-based student networks capable of providing an example-based explanation for their prediction.
We show that our prototype-based networks beyond similarity kernels deliver meaningful explanations and promising outlier detection results without compromising classification accuracy.
arXiv Detail & Related papers (2020-11-11T05:58:17Z) - Few-shot Visual Reasoning with Meta-analogical Contrastive Learning [141.2562447971]
We propose to solve a few-shot (or low-shot) visual reasoning problem, by resorting to analogical reasoning.
We extract structural relationships between elements in both domains, and enforce them to be as similar as possible with analogical learning.
We validate our method on RAVEN dataset, on which it outperforms state-of-the-art method, with larger gains when the training data is scarce.
arXiv Detail & Related papers (2020-07-23T14:00:34Z) - Using local dynamics to explain analog forecasting of chaotic systems [0.0]
This study investigates the properties of different analog forecasting strategies by taking local approximations of the system's dynamics.
We find that analog forecasting performances are highly linked to the local Jacobian matrix of the flow map.
The proposed methodology allows to estimate analog forecasting errors, and to compare different analog methods.
arXiv Detail & Related papers (2020-07-22T08:43:11Z) - Balance-Subsampled Stable Prediction [55.13512328954456]
We propose a novel balance-subsampled stable prediction (BSSP) algorithm based on the theory of fractional factorial design.
A design-theoretic analysis shows that the proposed method can reduce the confounding effects among predictors induced by the distribution shift.
Numerical experiments on both synthetic and real-world data sets demonstrate that our BSSP algorithm significantly outperforms the baseline methods for stable prediction across unknown test data.
arXiv Detail & Related papers (2020-06-08T07:01:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.