Dealing with Semantic Underspecification in Multimodal NLP
- URL: http://arxiv.org/abs/2306.05240v1
- Date: Thu, 8 Jun 2023 14:39:24 GMT
- Title: Dealing with Semantic Underspecification in Multimodal NLP
- Authors: Sandro Pezzelle
- Abstract summary: Intelligent systems that aim at mastering language as humans do must deal with its semantic underspecification.
Standard NLP models have, in principle, no or limited access to such extra information.
multimodal systems grounding language into other modalities, such as vision, are naturally equipped to account for this phenomenon.
- Score: 3.5846770619764423
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Intelligent systems that aim at mastering language as humans do must deal
with its semantic underspecification, namely, the possibility for a linguistic
signal to convey only part of the information needed for communication to
succeed. Consider the usages of the pronoun they, which can leave the gender
and number of its referent(s) underspecified. Semantic underspecification is
not a bug but a crucial language feature that boosts its storage and processing
efficiency. Indeed, human speakers can quickly and effortlessly integrate
semantically-underspecified linguistic signals with a wide range of
non-linguistic information, e.g., the multimodal context, social or cultural
conventions, and shared knowledge. Standard NLP models have, in principle, no
or limited access to such extra information, while multimodal systems grounding
language into other modalities, such as vision, are naturally equipped to
account for this phenomenon. However, we show that they struggle with it, which
could negatively affect their performance and lead to harmful consequences when
used for applications. In this position paper, we argue that our community
should be aware of semantic underspecification if it aims to develop language
technology that can successfully interact with human users. We discuss some
applications where mastering it is crucial and outline a few directions toward
achieving this goal.
Related papers
- A Taxonomy of Ambiguity Types for NLP [53.10379645698917]
We propose a taxonomy of ambiguity types as seen in English to facilitate NLP analysis.
Our taxonomy can help make meaningful splits in language ambiguity data, allowing for more fine-grained assessments of both datasets and model performance.
arXiv Detail & Related papers (2024-03-21T01:47:22Z) - Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language
Pretraining? [34.609984453754656]
We aim to elucidate the impact of comprehensive linguistic knowledge, including semantic expression and syntactic structure, on multimodal alignment.
Specifically, we design and release the SNARE, the first large-scale multimodal alignment probing benchmark.
arXiv Detail & Related papers (2023-08-24T16:17:40Z) - Uncertainty in Natural Language Generation: From Theory to Applications [42.55924708592451]
We argue that a principled treatment of uncertainty can assist in creating systems and evaluation protocols better aligned with these goals.
We first present the fundamental theory, frameworks and vocabulary required to represent uncertainty.
We then propose a two-dimensional taxonomy that is more informative and faithful than the popular aleatoric/epistemic dichotomy.
arXiv Detail & Related papers (2023-07-28T17:51:21Z) - Towards Bridging the Digital Language Divide [4.234367850767171]
multilingual language processing systems often exhibit a hardwired, yet usually involuntary and hidden representational preference towards certain languages.
We show that biased technology is often the result of research and development methodologies that do not do justice to the complexity of the languages being represented.
We present a new initiative that aims at reducing linguistic bias through both technological design and methodology.
arXiv Detail & Related papers (2023-07-25T10:53:20Z) - Reasoning over the Air: A Reasoning-based Implicit Semantic-Aware
Communication Framework [124.6509194665514]
A novel implicit semantic-aware communication (iSAC) architecture is proposed for representing, communicating, and interpreting the implicit semantic meaning between source and destination users.
A projection-based semantic encoder is proposed to convert the high-dimensional graphical representation of explicit semantics into a low-dimensional semantic constellation space for efficient physical channel transmission.
A generative adversarial imitation learning-based solution, called G-RML, is proposed to enable the destination user to learn and imitate the implicit semantic reasoning process of source user.
arXiv Detail & Related papers (2023-06-20T01:32:27Z) - Imitation Learning-based Implicit Semantic-aware Communication Networks:
Multi-layer Representation and Collaborative Reasoning [68.63380306259742]
Despite its promising potential, semantic communications and semantic-aware networking are still at their infancy.
We propose a novel reasoning-based implicit semantic-aware communication network architecture that allows multiple tiers of CDC and edge servers to collaborate.
We introduce a new multi-layer representation of semantic information taking into consideration both the hierarchical structure of implicit semantics as well as the personalized inference preference of individual users.
arXiv Detail & Related papers (2022-10-28T13:26:08Z) - An Empirical Revisiting of Linguistic Knowledge Fusion in Language
Understanding Tasks [33.765874588342285]
Infusing language models with syntactic or semantic knowledge from structural linguistic priors has shown improvements on many language understanding tasks.
We conduct empirical study of replacing parsed graphs or trees with trivial ones for tasks in the GLUE benchmark.
It reveals that the gains might not be significantly attributed to explicit linguistic priors but rather to more feature interactions brought by fusion layers.
arXiv Detail & Related papers (2022-10-24T07:47:32Z) - On the cross-lingual transferability of multilingual prototypical models
across NLU tasks [2.44288434255221]
Supervised deep learning-based approaches have been applied to task-oriented dialog and have proven to be effective for limited domain and language applications.
In practice, these approaches suffer from the drawbacks of domain-driven design and under-resourced languages.
This article proposes to investigate the cross-lingual transferability of using synergistically few-shot learning with prototypical neural networks and multilingual Transformers-based models.
arXiv Detail & Related papers (2022-07-19T09:55:04Z) - Towards Zero-shot Language Modeling [90.80124496312274]
We construct a neural model that is inductively biased towards learning human languages.
We infer this distribution from a sample of typologically diverse training languages.
We harness additional language-specific side information as distant supervision for held-out languages.
arXiv Detail & Related papers (2021-08-06T23:49:18Z) - Neural Abstructions: Abstractions that Support Construction for Grounded
Language Learning [69.1137074774244]
Leveraging language interactions effectively requires addressing limitations in the two most common approaches to language grounding.
We introduce the idea of neural abstructions: a set of constraints on the inference procedure of a label-conditioned generative model.
We show that with this method a user population is able to build a semantic modification for an open-ended house task in Minecraft.
arXiv Detail & Related papers (2021-07-20T07:01:15Z) - AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages
with Adversarial Examples [51.048234591165155]
We present AM2iCo, Adversarial and Multilingual Meaning in Context.
It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts.
Results reveal that current SotA pretrained encoders substantially lag behind human performance.
arXiv Detail & Related papers (2021-04-17T20:23:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.