NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly

• URL: http://arxiv.org/abs/2210.08604v2
• Date: Sat, 13 Jan 2024 22:46:01 GMT
• Title: NormSAGE: Multi-Lingual Multi-Cultural Norm Discovery from Conversations On-the-Fly
• Authors: Yi R. Fung, Tuhin Chakraborty, Hao Guo, Owen Rambow, Smaranda Muresan, Heng Ji
• Abstract summary: We introduce a framework for addressing the novel task of conversation-grounded multi-lingual, multi-cultural norm discovery. NormSAGE elicits knowledge about norms through directed questions representing the norm discovery task and conversation context. It further addresses the risk of language model hallucination with a self-verification mechanism ensuring that the norms discovered are correct.
• Score: 61.77957329364812
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Norm discovery is important for understanding and reasoning about the acceptable behaviors and potential violations in human communication and interactions. We introduce NormSage, a framework for addressing the novel task of conversation-grounded multi-lingual, multi-cultural norm discovery, based on language model prompting and self-verification. NormSAGE leverages the expressiveness and implicit knowledge of the pretrained GPT-3 language model backbone, to elicit knowledge about norms through directed questions representing the norm discovery task and conversation context. It further addresses the risk of language model hallucination with a self-verification mechanism ensuring that the norms discovered are correct and are substantially grounded to their source conversations. Evaluation results show that our approach discovers significantly more relevant and insightful norms for conversations on-the-fly compared to baselines (>10+% in Likert scale rating). The norms discovered from Chinese conversation are also comparable to the norms discovered from English conversation in terms of insightfulness and correctness (<3% difference). In addition, the culture-specific norms are promising quality, allowing for 80% accuracy in culture pair human identification. Finally, our grounding process in norm discovery self-verification can be extended for instantiating the adherence and violation of any norm for a given conversation on-the-fly, with explainability and transparency. NormSAGE achieves an AUC of 95.4% in grounding, with natural language explanation matching human-written quality.

Related papers

• Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning [84.94709351266557]
  We focus on the trustworthiness of language models with respect to retrieval augmentation. We deem that retrieval-augmented language models have the inherent capabilities of supplying response according to both contextual and parametric knowledge. Inspired by aligning language models with human preference, we take the first step towards aligning retrieval-augmented language models to a status where it responds relying merely on the external evidence.
  arXiv  Detail & Related papers  (2024-10-22T09:25:21Z)
• Normative Requirements Operationalization with Large Language Models [3.456725053685842]
  Normative non-functional requirements specify constraints that a system must observe in order to avoid violations of social, legal, ethical, empathetic, and cultural norms. Recent research has tackled this challenge using a domain-specific language to specify normative requirements. We propose a complementary approach that uses Large Language Models to extract semantic relationships between abstract representations of system capabilities.
  arXiv  Detail & Related papers  (2024-04-18T17:01:34Z)
• Detecting value-expressive text posts in Russian social media [0.0]
  We aimed to find a model that can accurately detect value-expressive posts in Russian social media VKontakte. A training dataset of 5,035 posts was annotated by three experts, 304 crowd-workers and ChatGPT. ChatGPT was more consistent but struggled with spam detection.
  arXiv  Detail & Related papers  (2023-12-14T14:18:27Z)
• NormDial: A Comparable Bilingual Synthetic Dialog Dataset for Modeling Social Norm Adherence and Violation [18.605252945314724]
  We present a high-quality dyadic dialogue dataset with turn-by-turn annotations of social norm adherences and violations for Chinese and American cultures. Our dataset is synthetically generated in both Chinese and English using a human-in-the-loop pipeline.
  arXiv  Detail & Related papers  (2023-10-23T04:38:34Z)
• NormMark: A Weakly Supervised Markov Model for Socio-cultural Norm Discovery [46.16583206206433]
  Existing methods for norm recognition tend to focus only on surface-level features of dialogues. We propose NormMark, a probabilistic generative Markov model to carry the latent features throughout a dialogue. We show that our approach achieves higher F1 score, outperforming current state-of-the-art methods, including GPT3.
  arXiv  Detail & Related papers  (2023-05-26T03:03:37Z)
• Natural Language Decompositions of Implicit Content Enable Better Text Representations [56.85319224208865]
  We introduce a method for the analysis of text that takes implicitly communicated content explicitly into account. We use a large language model to produce sets of propositions that are inferentially related to the text that has been observed. Our results suggest that modeling the meanings behind observed language, rather than the literal text alone, is a valuable direction for NLP.
  arXiv  Detail & Related papers  (2023-05-23T23:45:20Z)
• Language Models as Inductive Reasoners [125.99461874008703]
  We propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts. We create a dataset termed DEER containing 1.2k rule-fact pairs for the task, where rules and facts are written in natural language. We provide the first and comprehensive analysis of how well pretrained language models can induce natural language rules from natural language facts.
  arXiv  Detail & Related papers  (2022-12-21T11:12:14Z)
• On Reality and the Limits of Language Data: Aligning LLMs with Human Norms [10.02997544238235]
  Large Language Models (LLMs) harness linguistic associations in vast natural language data for practical applications. We explore this question using a novel and tightly controlled reasoning test (ART) and compare human norms against versions of GPT-3. Our findings highlight the categories of common-sense relations models that could learn directly from data and areas of weakness.
  arXiv  Detail & Related papers  (2022-08-25T10:21:23Z)
• Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? [62.74872383104381]
  We investigate the effectiveness of natural language interventions for reading-comprehension systems. We propose a new language understanding task, Linguistic Ethical Interventions (LEI), where the goal is to amend a question-answering (QA) model's unethical behavior.
  arXiv  Detail & Related papers  (2021-06-02T20:57:58Z)
• Unsupervised Commonsense Question Answering with Self-Talk [71.63983121558843]
  We propose an unsupervised framework based on self-talk as a novel alternative to commonsense tasks. Inspired by inquiry-based discovery learning, our approach inquires language models with a number of information seeking questions. Empirical results demonstrate that the self-talk procedure substantially improves the performance of zero-shot language model baselines.
  arXiv  Detail & Related papers  (2020-04-11T20:43:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.