YASO: A New Benchmark for Targeted Sentiment Analysis

• URL: http://arxiv.org/abs/2012.14541v1
• Date: Tue, 29 Dec 2020 00:25:15 GMT
• Title: YASO: A New Benchmark for Targeted Sentiment Analysis
• Authors: Matan Orbach, Orith Toledo-Ronen, Artem Spector, Ranit Aharonov, Yoav Katz and Noam Slonim
• Abstract summary: We present YASO -- a new crowd-sourced TSA evaluation dataset. The dataset contains 2,215 English sentences from movie, business and product reviews, and 7,415 terms and their sentiments annotated within these sentences. Our analysis verifies the reliability of our annotations, and explores the characteristics of the collected data.
• Score: 12.60266470026856
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sentiment analysis research has shifted over the years from the analysis of full documents or single sentences to a finer-level of detail -- identifying the sentiment towards single words or phrases -- with the task of Targeted Sentiment Analysis (TSA). While this problem is attracting a plethora of works focusing on algorithmic aspects, they are typically evaluated on a selection from a handful of datasets, and little effort, if any, is dedicated to the expansion of the available evaluation data. In this work, we present YASO -- a new crowd-sourced TSA evaluation dataset, collected using a new annotation scheme for labeling targets and their sentiments. The dataset contains 2,215 English sentences from movie, business and product reviews, and 7,415 terms and their corresponding sentiments annotated within these sentences. Our analysis verifies the reliability of our annotations, and explores the characteristics of the collected data. Lastly, benchmark results using five contemporary TSA systems lay the foundation for future work, and show there is ample room for improvement on this challenging new dataset.

Related papers

• Wiki-TabNER:Advancing Table Interpretation Through Named Entity Recognition [19.423556742293762]
  We analyse a widely used benchmark dataset for evaluation of TI tasks. To overcome this drawback, we construct and annotate a new more challenging dataset. We propose a prompting framework for evaluating the newly developed large language models.
  arXiv  Detail & Related papers  (2024-03-07T15:22:07Z)
• Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries [67.0083902913112]
  We develop the Text2Analysis benchmark, incorporating advanced analysis tasks. We also develop five innovative and effective annotation methods. We evaluate five state-of-the-art models using three different metrics.
  arXiv  Detail & Related papers  (2023-12-21T08:50:41Z)
• SOUL: Towards Sentiment and Opinion Understanding of Language [96.74878032417054]
  We propose a new task called Sentiment and Opinion Understanding of Language (SOUL) SOUL aims to evaluate sentiment understanding through two subtasks: Review (RC) and Justification Generation (JG)
  arXiv  Detail & Related papers  (2023-10-27T06:48:48Z)
• OATS: Opinion Aspect Target Sentiment Quadruple Extraction Dataset for Aspect-Based Sentiment Analysis [55.61047894397937]
  Aspect-based sentiment analysis (ABSA) delves into understanding sentiments specific to distinct elements within a user-generated review. We introduce the OATS dataset, which encompasses three fresh domains and consists of 27,470 sentence-level quadruples and 17,092 review-levels. Our initiative seeks to bridge specific observed gaps: the recurrent focus on familiar domains like restaurants and laptops, limited data for intricate quadruple extraction tasks, and an occasional oversight of the synergy between sentence and review-level sentiments.
  arXiv  Detail & Related papers  (2023-09-23T07:39:16Z)
• Entity-Level Sentiment Analysis (ELSA): An exploratory task survey [2.191505742658975]
  This paper explores the task of identifying the overall sentiment expressed towards volitional entities in a document. We show that sentiment in our dataset is expressed not only with an entity mention as target, but also towards targets with a sentiment-relevant relation to a volitional entity.
  arXiv  Detail & Related papers  (2023-04-27T15:01:20Z)
• DiaASQ : A Benchmark of Conversational Aspect-based Sentiment Quadruple Analysis [84.80347062834517]
  We introduce DiaASQ, aiming to detect the quadruple of target-aspect-opinion-sentiment in a dialogue. We manually construct a large-scale high-quality DiaASQ dataset in both Chinese and English languages. We develop a neural model to benchmark the task, which advances in effectively performing end-to-end quadruple prediction.
  arXiv  Detail & Related papers  (2022-11-10T17:18:20Z)
• Survey of Aspect-based Sentiment Analysis Datasets [55.61047894397937]
  Aspect-based sentiment analysis (ABSA) is a natural language processing problem that requires analyzing user-generated reviews. Numerous yet scattered corpora for ABSA make it difficult for researchers to identify corpora best suited for a specific ABSA subtask quickly. This study aims to present a database of corpora that can be used to train and assess autonomous ABSA systems.
  arXiv  Detail & Related papers  (2022-04-11T16:23:36Z)
• SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech [44.68649535280397]
  We propose a suite of benchmark tasks for Spoken Language Understanding Evaluation (SLUE) SLUE consists of limited-size labeled training sets and corresponding evaluation sets. We present the first phase of the SLUE benchmark suite, consisting of named entity recognition, sentiment analysis, and ASR on the corresponding datasets. We provide new transcriptions and annotations on subsets of the VoxCeleb and VoxPopuli datasets, evaluation metrics and results for baseline models, and an open-source toolkit to reproduce the baselines and evaluate new models.
  arXiv  Detail & Related papers  (2021-11-19T18:59:23Z)
• Exploring Conditional Text Generation for Aspect-Based Sentiment Analysis [28.766801337922306]
  Aspect-based sentiment analysis (ABSA) is an NLP task that entails processing user-generated reviews to determine (i) the target being evaluated, (ii) the aspect category to which it belongs, and (iii) the sentiment expressed towards the target and aspect pair. We propose transforming ABSA into an abstract summary-like conditional text generation task that uses targets, aspects, and polarities to generate auxiliary statements.
  arXiv  Detail & Related papers  (2021-10-05T20:08:25Z)
• Author's Sentiment Prediction [13.459029439420872]
  PerSenT is a dataset of crowd-sourced annotations of the sentiment expressed by the authors towards the main entities in news articles. The dataset also includes paragraph-level sentiment annotations to provide more fine-grained supervision for the task. We release this dataset with 5.3k documents and 38k paragraphs covering 3.2k unique entities as a challenge in entity sentiment analysis.
  arXiv  Detail & Related papers  (2020-11-12T00:03:26Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.