Entity Embeddings : Perspectives Towards an Omni-Modality Era for Large Language Models

• URL: http://arxiv.org/abs/2310.18390v1
• Date: Fri, 27 Oct 2023 17:04:10 GMT
• Title: Entity Embeddings : Perspectives Towards an Omni-Modality Era for Large Language Models
• Authors: Eren Unlu, Unver Ciftci
• Abstract summary: Large Language Models (LLMs) are evolving to integrate multiple modalities, such as text, image, and audio into a unified linguistic space. We envision a future direction where conceptual entities defined in sequences of text can also be imagined as modalities.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are evolving to integrate multiple modalities, such as text, image, and audio into a unified linguistic space. We envision a future direction based on this framework where conceptual entities defined in sequences of text can also be imagined as modalities. Such a formulation has the potential to overcome the cognitive and computational limitations of current models. Several illustrative examples of such potential implicit modalities are given. Along with vast promises of the hypothesized structure, expected challenges are discussed as well.

Related papers

• Predictive Simultaneous Interpretation: Harnessing Large Language Models for Democratizing Real-Time Multilingual Communication [0.0]
  We present a novel algorithm that generates real-time translations by predicting speaker utterances and expanding multiple possibilities in a tree-like structure. Our theoretical analysis, supported by illustrative examples, suggests that this approach could lead to more natural and fluent translations with minimal latency.
  arXiv  Detail & Related papers  (2024-07-02T13:18:28Z)
• DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment [82.86363991170546]
  We propose a Descriptive Speech-Text Alignment approach that leverages speech captioning to bridge the gap between speech and text modalities. Our model demonstrates superior performance on the Dynamic-SUPERB benchmark, particularly in generalizing to unseen tasks. These findings highlight the potential to reshape instruction-following SLMs by incorporating descriptive rich, speech captions.
  arXiv  Detail & Related papers  (2024-06-27T03:52:35Z)
• Concept Formation and Alignment in Language Models: Bridging Statistical Patterns in Latent Space to Concept Taxonomy [11.232704182001253]
  This paper explores the concept formation and alignment within the realm of language models (LMs) We propose a mechanism for identifying concepts and their hierarchical organization within the semantic representations learned by various LMs.
  arXiv  Detail & Related papers  (2024-06-08T01:27:19Z)
• Explaining Multi-modal Large Language Models by Analyzing their Vision Perception [4.597864989500202]
  This research proposes a novel approach to enhance the interpretability of MLLMs by focusing on the image embedding component. We combine an open-world localization model with a MLLM, thus creating a new architecture able to simultaneously produce text and object localization outputs from the same vision embedding.
  arXiv  Detail & Related papers  (2024-05-23T14:24:23Z)
• How Well Do Text Embedding Models Understand Syntax? [50.440590035493074]
  The ability of text embedding models to generalize across a wide range of syntactic contexts remains under-explored. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges. We propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios.
  arXiv  Detail & Related papers  (2023-11-14T08:51:00Z)
• RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models [57.12888828853409]
  RAVEN is a model that combines retrieval-augmented masked language modeling and prefix language modeling. Fusion-in-Context Learning enables the model to leverage more in-context examples without requiring additional training. Our work underscores the potential of retrieval-augmented encoder-decoder language models for in-context learning.
  arXiv  Detail & Related papers  (2023-08-15T17:59:18Z)
• Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
  Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
  arXiv  Detail & Related papers  (2023-07-25T17:59:18Z)
• SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
  We propose a framework to enable in-context learning in large language models. A meta-model can learn on self-supervised prompts consisting of tailored demonstrations. Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
  arXiv  Detail & Related papers  (2023-07-15T08:33:08Z)
• Improving Factuality and Reasoning in Language Models through Multiagent Debate [95.10641301155232]
  We present a complementary approach to improve language responses where multiple language model instances propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer. Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks. Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate.
  arXiv  Detail & Related papers  (2023-05-23T17:55:11Z)
• Modelling Compositionality and Structure Dependence in Natural Language [0.12183405753834563]
  Drawing on linguistics and set theory, a formalisation of these ideas is presented in the first half of this thesis. We see how cognitive systems that process language need to have certain functional constraints. Using the advances of word embedding techniques, a model of relational learning is simulated.
  arXiv  Detail & Related papers  (2020-11-22T17:28:50Z)
• What are the Goals of Distributional Semantics? [12.640283469603355]
  I take a broad linguistic perspective, looking at how well current models can deal with various semantic challenges. I conclude that, while linguistic insights can guide the design of model architectures, future progress will require balancing the often conflicting demands of linguistic expressiveness and computational tractability.
  arXiv  Detail & Related papers  (2020-05-06T17:36:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.