Evaluating LLMs Capabilities Towards Understanding Social Dynamics

• URL: http://arxiv.org/abs/2411.13008v1
• Date: Wed, 20 Nov 2024 03:16:07 GMT
• Title: Evaluating LLMs Capabilities Towards Understanding Social Dynamics
• Authors: Anique Tahir, Lu Cheng, Manuel Sandoval, Yasin N. Silva, Deborah L. Hall, Huan Liu,
• Abstract summary: Generative Models, such as Llama and ChatGPT, have recently exploded in popularity due to their capabilities in zero-shot question-answering. This work provides a critical analysis regarding generative LLM's ability to understand language and dynamics in social contexts.
• Score: 16.189101260093064
• License:
• Abstract: Social media discourse involves people from different backgrounds, beliefs, and motives. Thus, often such discourse can devolve into toxic interactions. Generative Models, such as Llama and ChatGPT, have recently exploded in popularity due to their capabilities in zero-shot question-answering. Because these models are increasingly being used to ask questions of social significance, a crucial research question is whether they can understand social media dynamics. This work provides a critical analysis regarding generative LLM's ability to understand language and dynamics in social contexts, particularly considering cyberbullying and anti-cyberbullying (posts aimed at reducing cyberbullying) interactions. Specifically, we compare and contrast the capabilities of different large language models (LLMs) to understand three key aspects of social dynamics: language, directionality, and the occurrence of bullying/anti-bullying messages. We found that while fine-tuned LLMs exhibit promising results in some social media understanding tasks (understanding directionality), they presented mixed results in others (proper paraphrasing and bullying/anti-bullying detection). We also found that fine-tuning and prompt engineering mechanisms can have positive effects in some tasks. We believe that a understanding of LLM's capabilities is crucial to design future models that can be effectively used in social applications.

Related papers

• Engagement-Driven Content Generation with Large Language Models [8.049552839071918]
  Large Language Models (LLMs) exhibit significant persuasion capabilities in one-on-one interactions. This study investigates the potential social impact of LLMs in interconnected users and complex opinion dynamics.
  arXiv  Detail & Related papers  (2024-11-20T10:40:08Z)
• Persuasion with Large Language Models: a Survey [49.86930318312291]
  Large Language Models (LLMs) have created new disruptive possibilities for persuasive communication. In areas such as politics, marketing, public health, e-commerce, and charitable giving, such LLM Systems have already achieved human-level or even super-human persuasiveness. Our survey suggests that the current and future potential of LLM-based persuasion poses profound ethical and societal risks.
  arXiv  Detail & Related papers  (2024-11-11T10:05:52Z)
• Large Language Models Reflect the Ideology of their Creators [73.25935570218375]
  Large language models (LLMs) are trained on vast amounts of data to generate natural language. We uncover notable diversity in the ideological stance exhibited across different LLMs and languages.
  arXiv  Detail & Related papers  (2024-10-24T04:02:30Z)
• I Want to Break Free! Persuasion and Anti-Social Behavior of LLMs in Multi-Agent Settings with Social Hierarchy [13.68625980741047]
  We study interaction patterns of Large Language Model (LLM)-based agents in a context characterized by strict social hierarchy. We study two types of phenomena: persuasion and anti-social behavior in simulated scenarios involving a guard and a prisoner agent.
  arXiv  Detail & Related papers  (2024-10-09T17:45:47Z)
• AgentPeerTalk: Empowering Students through Agentic-AI-Driven Discernment of Bullying and Joking in Peer Interactions in Schools [0.0]
  This study examined the potential of large language models (LLMs) to empower students by discerning between bullying and joking in school peer interactions. ChatGPT-4 excelled in context-specific accuracy after implementing the agentic approach.
  arXiv  Detail & Related papers  (2024-07-27T05:50:02Z)
• SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents [73.35393511272791]
  We propose an interactive learning method, SOTOPIA-$pi$, improving the social intelligence of language agents. This method leverages behavior cloning and self-reinforcement training on filtered social interaction data according to large language model (LLM) ratings.
  arXiv  Detail & Related papers  (2024-03-13T17:17:48Z)
• SoMeLVLM: A Large Vision Language Model for Social Media Processing [78.47310657638567]
  We introduce a Large Vision Language Model for Social Media Processing (SoMeLVLM) SoMeLVLM is a cognitive framework equipped with five key capabilities including knowledge & comprehension, application, analysis, evaluation, and creation. Our experiments demonstrate that SoMeLVLM achieves state-of-the-art performance in multiple social media tasks.
  arXiv  Detail & Related papers  (2024-02-20T14:02:45Z)
• Think Before You Speak: Cultivating Communication Skills of Large Language Models via Inner Monologue [73.69510478736483]
  Large language models (LLMs) can generate fluent, coherent, and diverse responses. However, they lack a crucial ability: communication skills. This article aims to empower LLMs with communication skills through inner monologues. Experimental results show that the proposed CSIM strategy improves the backbone models and outperforms the baselines.
  arXiv  Detail & Related papers  (2023-11-13T16:19:42Z)
• Qualitative Analysis of a Graph Transformer Approach to Addressing Hate Speech: Adapting to Dynamically Changing Content [8.393770595114763]
  We offer a detailed qualitative analysis of this solution for hate speech detection in social networks. A key insight is that the focus on reasoning about the concept of context positions us well to be able to support multi-modal analysis of online posts. We conclude with a reflection on how the problem we are addressing relates especially well to the theme of dynamic change.
  arXiv  Detail & Related papers  (2023-01-25T23:32:32Z)
• Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational Agents [60.27066549589362]
  Social language used by human agents is associated with greater users' responsiveness and task completion. The model uses a sequence-to-sequence deep learning architecture, extended with a social language understanding element. Evaluation in terms of content preservation and social language level using both human judgment and automatic linguistic measures shows that the model can generate responses that enable agents to address users' issues in a more socially appropriate way.
  arXiv  Detail & Related papers  (2020-12-29T08:22:48Z)
• Aggressive, Repetitive, Intentional, Visible, and Imbalanced: Refining Representations for Cyberbullying Classification [4.945634077636197]
  We study the nuanced problem of cyberbullying using five explicit factors to represent its social and linguistic aspects. These results demonstrate the importance of representing and modeling cyberbullying as a social phenomenon.
  arXiv  Detail & Related papers  (2020-04-04T00:35:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.