Annotation Quality in Aspect-Based Sentiment Analysis: A Case Study Comparing Experts, Students, Crowdworkers, and Large Language Model

• URL: http://arxiv.org/abs/2605.03624v1
• Date: Tue, 05 May 2026 10:54:05 GMT
• Title: Annotation Quality in Aspect-Based Sentiment Analysis: A Case Study Comparing Experts, Students, Crowdworkers, and Large Language Model
• Abstract summary: Aspect-Based Sentiment Analysis (ABSA) enables fine-grained opinion analysis by identifying sentiments toward specific aspects or targets within a text.<n>While ABSA has been widely studied for English, research on other languages such as German remains limited, largely due to the lack of high-quality annotated datasets.<n>This paper examines how different annotation sources influence the development of German ABSA.
• Score: 5.054478273897118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aspect-Based Sentiment Analysis (ABSA) enables fine-grained opinion analysis by identifying sentiments toward specific aspects or targets within a text. While ABSA has been widely studied for English, research on other languages such as German remains limited, largely due to the lack of high-quality annotated datasets. This paper examines how different annotation sources influence the development of German ABSA. To this end, an existing dataset is re-annotated by experts to establish a ground truth, which serves as a reference for evaluating annotations produced by students, crowdworkers, Large Language Models (LLMs), and experts. Annotation quality is compared using Inter-Annotator Agreement (IAA) and its impact on downstream model performance for different ABSA subtasks. The evaluation focuses on Aspect Category Sentiment Analysis (ACSA) and Target Aspect Sentiment Detection (TASD). We apply State-of-the-Art (SOTA) methods for ABSA, including BERT-, T5-, and LLaMA-based approaches to assess performance differences, spanning fine-tuning and in-context learning with instruction prompts. The findings provide practical insights into trade-offs between annotation reliability and efficiency, offering guidance for dataset construction in under-resourced Natural Language Processing (NLP) scenarios.

Related papers

• SSP-based construction of evaluation-annotated data for fine-grained aspect-based sentiment analysis [0.0]
  We report the construction of a Korean evaluation-annotated corpus.<n>We use it to cover e-commerce reviews containing sentiment and non-sentiment linguistic patterns.<n>For evaluation, the KoBERT and KcBERT models are trained on the annotated dataset, showing robust performances of F1 0.88 and F1 0.90.
  arXiv  Detail & Related papers  (2026-05-08T08:52:37Z)
• DimABSA: Building Multilingual and Multidomain Datasets for Dimensional Aspect-Based Sentiment Analysis [57.70022214686838]
  DimABSA is the first multilingual, dimensional ABSA resource annotated with both traditional ABSA elements and VA scores.<n>This resource contains 76,958 aspect instances across 42,590 sentences, spanning six languages and four domains.
  arXiv  Detail & Related papers  (2026-01-30T14:30:35Z)
• M-ABSA: A Multilingual Dataset for Aspect-Based Sentiment Analysis [34.298472723655344]
  We present M-ABSA, a comprehensive dataset spanning 7 domains and 21 languages.<n>Our primary focus is on triplet extraction, which involves identifying aspect terms, aspect categories, and sentiment polarities.<n>Our empirical findings highlight that the dataset enables diverse evaluation tasks, such as multilingual and multi-domain transfer learning.
  arXiv  Detail & Related papers  (2025-02-17T14:16:01Z)
• Iterative Data Generation with Large Language Models for Aspect-based Sentiment Analysis [39.57537769578304]
  We propose a systematic Iterative Data Generation framework, namely IDG, to boost the performance of ABSA. The core of IDG is to make full use of the powerful abilities (i.e., instruction-following, in-context learning and self-reflection) of LLMs to iteratively generate more fluent and diverse pseudo-label data. IDG brings consistent and significant performance gains among five baseline ABSA models.
  arXiv  Detail & Related papers  (2024-06-29T07:00:37Z)
• ROAST: Review-level Opinion Aspect Sentiment Target Joint Detection for ABSA [50.90538760832107]
  This research presents a novel task, Review-Level Opinion Aspect Sentiment Target (ROAST) ROAST seeks to close the gap between sentence-level and text-level ABSA by identifying every ABSA constituent at the review level. We extend the available datasets to enable ROAST, addressing the drawbacks noted in previous research.
  arXiv  Detail & Related papers  (2024-05-30T17:29:15Z)
• A Hybrid Approach To Aspect Based Sentiment Analysis Using Transfer Learning [3.30307212568497]
  We propose a hybrid approach for Aspect Based Sentiment Analysis using transfer learning. The approach focuses on generating weakly-supervised annotations by exploiting the strengths of both large language models (LLM) and traditional syntactic dependencies.
  arXiv  Detail & Related papers  (2024-03-25T23:02:33Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• Survey of Aspect-based Sentiment Analysis Datasets [55.61047894397937]
  Aspect-based sentiment analysis (ABSA) is a natural language processing problem that requires analyzing user-generated reviews. Numerous yet scattered corpora for ABSA make it difficult for researchers to identify corpora best suited for a specific ABSA subtask quickly. This study aims to present a database of corpora that can be used to train and assess autonomous ABSA systems.
  arXiv  Detail & Related papers  (2022-04-11T16:23:36Z)
• BERT-ASC: Auxiliary-Sentence Construction for Implicit Aspect Learning in Sentiment Analysis [4.522719296659495]
  This paper proposes a unified framework to address aspect categorization and aspect-based sentiment subtasks. We introduce a mechanism to construct an auxiliary-sentence for the implicit aspect using the corpus's semantic information. We then encourage BERT to learn aspect-specific representation in response to this auxiliary-sentence, not the aspect itself.
  arXiv  Detail & Related papers  (2022-03-22T13:12:27Z)
• Understanding Pre-trained BERT for Aspect-based Sentiment Analysis [71.40586258509394]
  This paper analyzes the pre-trained hidden representations learned from reviews on BERT for tasks in aspect-based sentiment analysis (ABSA) It is not clear how the general proxy task of (masked) language model trained on unlabeled corpus without annotations of aspects or opinions can provide important features for downstream tasks in ABSA.
  arXiv  Detail & Related papers  (2020-10-31T02:21:43Z)
• Weakly-Supervised Aspect-Based Sentiment Analysis via Joint Aspect-Sentiment Topic Embedding [71.2260967797055]
  We propose a weakly-supervised approach for aspect-based sentiment analysis. We learn sentiment, aspect> joint topic embeddings in the word embedding space. We then use neural models to generalize the word-level discriminative information.
  arXiv  Detail & Related papers  (2020-10-13T21:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.