Fine-Grained Detection of Solidarity for Women and Migrants in 155 Years of German Parliamentary Debates

• URL: http://arxiv.org/abs/2210.04359v2
• Date: Mon, 24 Jun 2024 20:01:19 GMT
• Title: Fine-Grained Detection of Solidarity for Women and Migrants in 155 Years of German Parliamentary Debates
• Authors: Aida Kostikova, Benjamin Paassen, Dominik Beese, Ole Pütz, Gregor Wiedemann, Steffen Eger,
• Abstract summary: We study solidarity towards women and migrants in German parliamentary debates between 1867 and 2022. Using 2,864 manually annotated text snippets, we evaluate large language models (LLMs) like Llama 3, GPT-3.5, and GPT-4. Using GPT-4, we automatically annotate more than 18k further instances (with a cost of around 500 Euro) across 155 years.
• Score: 16.382860418871804
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Solidarity is a crucial concept to understand social relations in societies. In this paper, we explore fine-grained solidarity frames to study solidarity towards women and migrants in German parliamentary debates between 1867 and 2022. Using 2,864 manually annotated text snippets (with a cost exceeding 18k Euro), we evaluate large language models (LLMs) like Llama 3, GPT-3.5, and GPT-4. We find that GPT-4 outperforms other LLMs, approaching human annotation quality. Using GPT-4, we automatically annotate more than 18k further instances (with a cost of around 500 Euro) across 155 years and find that solidarity with migrants outweighs anti-solidarity but that frequencies and solidarity types shift over time. Most importantly, group-based notions of (anti-)solidarity fade in favor of compassionate solidarity, focusing on the vulnerability of migrant groups, and exchange-based anti-solidarity, focusing on the lack of (economic) contribution. Our study highlights the interplay of historical events, socio-economic needs, and political ideologies in shaping migration discourse and social cohesion. We also show that powerful LLMs, if carefully prompted, can be cost-effective alternatives to human annotation for hard social scientific tasks.

Related papers

• SS-Bench: A Benchmark for Social Story Generation and Evaluation [53.39177041545863]
  Children with Autism Spectrum Disorder (ASD) often misunderstand social situations and struggle to participate in daily routines. Social Stories are costly in creation and often limited in diversity and timeliness. We propose textbfSS-Bench, a textbfSocial textbfStory textbfBenchmark for generating and evaluating Social Stories.
  arXiv  Detail & Related papers  (2024-06-22T00:14:48Z)
• Annotation alignment: Comparing LLM and human annotations of conversational safety [10.143093546513857]
  We show that GPT-4 achieves a Pearson correlation of $r = 0.59$ with the average annotator rating, higher than the median annotator's correlation with the average. There is substantial idiosyncratic variation in correlation *within* groups, suggesting that race & gender do not fully capture differences in alignment.
  arXiv  Detail & Related papers  (2024-06-10T15:30:13Z)
• In Silico Sociology: Forecasting COVID-19 Polarization with Large Language Models [0.0]
  We train deep neural networks on massive archives of digitized text to learn the complex linguistic patterns that constitute historic and contemporary discourses. We reconstruct the public opinion landscape of 2019 to examine the extent to which the future polarization over COVID-19 was prefigured in existing political discourse. We find that the simulated respondents reproduce observed partisan differences in COVID-19 attitudes in 84% of cases, significantly greater than chance.
  arXiv  Detail & Related papers  (2024-05-23T22:10:12Z)
• The Call for Socially Aware Language Technologies [94.6762219597438]
  We argue that many of these issues share a common core: a lack of awareness of the factors, context, and implications of the social environment in which NLP operates. We argue that substantial challenges remain for NLP to develop social awareness and that we are just at the beginning of a new era for the field.
  arXiv  Detail & Related papers  (2024-05-03T18:12:39Z)
• AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic [0.0]
  We introduce AraTrust, the first comprehensive trustworthiness benchmark for Large Language Models (LLMs) in Arabic. GPT-4 was the most trustworthy LLM, while open-source models, particularly AceGPT 7B and Jais 13B, struggled to achieve a score of 60% in our benchmark.
  arXiv  Detail & Related papers  (2024-03-14T00:45:24Z)
• Navigating Multidimensional Ideologies with Reddit's Political Compass: Economic Conflict and Social Affinity [3.850666668546735]
  We analyze social interactions on Reddit under the lens of a multi-dimensional ideological framework: the political compass. By leveraging their self-declarations, we disentangle the ideological dimensions of users into economic (left-right) and social (libertarian richness-authoritarian) axes. We find significant homophily for interactions along the social axis of the political compass and demographic attributes. In contrast, we uncover a significant heterophily along the economic axis: left/right interactions exceed expectations by 10%.
  arXiv  Detail & Related papers  (2024-01-24T18:49:19Z)
• Branch-Solve-Merge Improves Large Language Model Evaluation and Generation [136.7876524839751]
  Large Language Models (LLMs) are frequently used for multi-faceted language generation and evaluation tasks. We propose Branch-Merge (BSM), a Large Language Model program (Schlag et al., 2023) for tackling such challenging natural language tasks. BSM improves the evaluation correctness and consistency for each LLM by enhancing human-LLM agreement by up to 26%.
  arXiv  Detail & Related papers  (2023-10-23T17:29:48Z)
• Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization [66.08074487429477]
  We investigate the stability and reliability of large language models (LLMs) as automatic evaluators for abstractive summarization. We find that while ChatGPT and GPT-4 outperform the commonly used automatic metrics, they are not ready as human replacements.
  arXiv  Detail & Related papers  (2023-05-22T14:58:13Z)
• Consistency Analysis of ChatGPT [65.268245109828]
  This paper investigates the trustworthiness of ChatGPT and GPT-4 regarding logically consistent behaviour. Our findings suggest that while both models appear to show an enhanced language understanding and reasoning ability, they still frequently fall short of generating logically consistent predictions.
  arXiv  Detail & Related papers  (2023-03-11T01:19:01Z)
• Changes in European Solidarity Before and During COVID-19: Evidence from a Large Crowd- and Expert-Annotated Twitter Dataset [77.27709662210363]
  We introduce the well-established social scientific concept of social solidarity and its contestation, anti-solidarity, as a new problem setting to supervised machine learning in NLP. We annotate 2.3k English and German tweets for (anti-)solidarity expressions, utilizing multiple human annotators and two annotation approaches (experts vs. crowds) Our results show that solidarity became increasingly salient and contested during the COVID-19 crisis.
  arXiv  Detail & Related papers  (2021-08-02T17:03:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.