MASSIVE Multilingual Abstract Meaning Representation: A Dataset and Baselines for Hallucination Detection

• URL: http://arxiv.org/abs/2405.19285v1
• Date: Wed, 29 May 2024 17:17:22 GMT
• Title: MASSIVE Multilingual Abstract Meaning Representation: A Dataset and Baselines for Hallucination Detection
• Authors: Michael Regan, Shira Wein, George Baker, Emilio Monti,
• Abstract summary: We introduce MASSIVE-AMR, a dataset with more than 84,000 text-to-graph annotations. AMR graphs for 1,685 information-seeking utterances mapped to 50+ typologically diverse languages. Results shed light on persistent issues using LLMs for structured parsing.
• Score: 3.6811136816751513
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Abstract Meaning Representation (AMR) is a semantic formalism that captures the core meaning of an utterance. There has been substantial work developing AMR corpora in English and more recently across languages, though the limited size of existing datasets and the cost of collecting more annotations are prohibitive. With both engineering and scientific questions in mind, we introduce MASSIVE-AMR, a dataset with more than 84,000 text-to-graph annotations, currently the largest and most diverse of its kind: AMR graphs for 1,685 information-seeking utterances mapped to 50+ typologically diverse languages. We describe how we built our resource and its unique features before reporting on experiments using large language models for multilingual AMR and SPARQL parsing as well as applying AMRs for hallucination detection in the context of knowledge base question answering, with results shedding light on persistent issues using LLMs for structured parsing.

Related papers

• FASSILA: A Corpus for Algerian Dialect Fake News Detection and Sentiment Analysis [0.0]
  The Algerian dialect (AD) faces challenges due to the absence of annotated corpora. This study outlines the development process of a specialized corpus for Fake News (FN) detection and sentiment analysis (SA) in AD called FASSILA.
  arXiv  Detail & Related papers  (2024-11-07T10:39:10Z)
• Analyzing the Role of Semantic Representations in the Era of Large Language Models [104.18157036880287]
  We investigate the role of semantic representations in the era of large language models (LLMs) We propose an AMR-driven chain-of-thought prompting method, which we call AMRCoT. We find that it is difficult to predict which input examples AMR may help or hurt on, but errors tend to arise with multi-word expressions.
  arXiv  Detail & Related papers  (2024-05-02T17:32:59Z)
• INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning [59.07490387145391]
  Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Their application to information retrieval (IR) tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. We introduce a novel instruction tuning dataset, INTERS, encompassing 20 tasks across three fundamental IR categories.
  arXiv  Detail & Related papers  (2024-01-12T12:10:28Z)
• "You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation [60.863629647985526]
  We examine the successes and limitations of the GPT-3, ChatGPT, and GPT-4 models in analysis of sentence meaning structure. We find that models can reliably reproduce the basic format of AMR, and can often capture core event, argument, and modifier structure. Overall, our findings indicate that these models out-of-the-box can capture aspects of semantic structure, but there remain key limitations in their ability to support fully accurate semantic analyses or parses.
  arXiv  Detail & Related papers  (2023-10-26T21:47:59Z)
• Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation [70.58243648754507]
  We introduce a new method to improve existing multilingual sentence embeddings with Abstract Meaning Representation (AMR) Compared with the original textual input, AMR is a structured semantic representation that presents the core concepts and relations in a sentence explicitly and unambiguously. Experiment results show that retrofitting multilingual sentence embeddings with AMR leads to better state-of-the-art performance on both semantic similarity and transfer tasks.
  arXiv  Detail & Related papers  (2022-10-18T11:37:36Z)
• A Simple and Effective Method To Eliminate the Self Language Bias in Multilingual Representations [7.571549274473274]
  Language agnostic and semantic-language information isolation is an emerging research direction for multilingual representations models. "Language Information Removal (LIR)" factors out language identity information from semantic related components in multilingual representations pre-trained on multi-monolingual data. LIR reveals that for weak-alignment multilingual systems, the principal components of semantic spaces primarily encodes language identity information.
  arXiv  Detail & Related papers  (2021-09-10T08:15:37Z)
• Translate, then Parse! A strong baseline for Cross-Lingual AMR Parsing [10.495114898741205]
  We develop models that project sentences from various languages onto their AMRs to capture their essential semantic structures. In this paper, we revisit a simple two-step base-line, and enhance it with a strong NMT system and a strong AMR. Our experiments show that T+P outperforms a recent state-of-the-art system across all tested languages.
  arXiv  Detail & Related papers  (2021-06-08T17:52:48Z)
• DiS-ReX: A Multilingual Dataset for Distantly Supervised Relation Extraction [15.649929244635269]
  We propose a new dataset, DiS-ReX, which alleviates these issues. Our dataset has more than 1.5 million sentences, spanning across 4 languages with 36 relation classes + 1 no relation (NA) class. We also modify the widely used bag attention models by encoding sentences using mBERT and provide the first benchmark results on multilingual DS-RE.
  arXiv  Detail & Related papers  (2021-04-17T22:44:38Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding [75.03682706791389]
  We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset. RxR is multilingual (English, Hindi, and Telugu) and larger (more paths and instructions) than other VLN datasets. It emphasizes the role of language in VLN by addressing known biases in paths and eliciting more references to visible entities.
  arXiv  Detail & Related papers  (2020-10-15T18:01:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.