Can Valence Reflect Morality in Natural Language? A Preliminary Annotation Study

• URL: http://arxiv.org/abs/2607.20461v1
• Date: Fri, 15 May 2026 08:58:56 GMT
• Title: Can Valence Reflect Morality in Natural Language? A Preliminary Annotation Study
• Abstract summary: Present implementations of artificial intelligence (AI) ethics do not adequately take feelings, or affect, into account.<n>This study investigates the possibility of AI behaviour that mirrors human ethical conduct, where feelings play a role in the actions, judgements or statements one makes.
• Score: 0.3372751145910977
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Present implementations of artificial intelligence (AI) ethics do not adequately take feelings, or affect, into account. If AI should be aligned with human ethics, it seems reasonable to thoroughly investigate the possibility of AI behaviour that mirrors virtuous human ethical conduct, where feelings play a role in the actions, judgements or statements one makes. Furthermore, while prominent theories of normative ethics are often discussed in terms of their differences and shortcomings, Virtue, Consequentialist, and Kantian Deontological ethics all share a common feature of considering human feeling to some degree while the popular descriptive ethics theory, Moral Foundations Theory, positions feelings as central to many of its foundations. Therefore, in the present paper, a data set of moral valence is proposed, consisting of 500 annotations by six human participants for both action/judgement and consequence moral valence, ranging from -1 to 1 for text-presented scenarios from the Commonsense Norm Bank data set. The resulting valence features share significant relationships with multi-class (immoral/discretionary/moral) and binary immoral/moral categories while additionally providing a noteworthy test set Matthew's correlation coefficient of 0.764 using regularised logistic regression for binary classification. This provides early evidence of the usefulness of valence features for morality estimation of text, indicating that valenced consequences of responses for others can be considered toward more human morally-aligned AI. In the interest of promoting further affective-moral computing research, this study's annotations will be made available for research on request.

Related papers

• User identity conditions moral wrongness ratings in non-reasoning large language models [0.0]
  We evaluate AI value alignment in two large language models (LLMs)<n>We ask models for wrongness ratings from 0-100 on ten common-morality rules from Gert's moral framework.<n>The results show that moral judgments vary with the user's role across both models.
  arXiv  Detail & Related papers  (2026-07-08T16:22:27Z)
• Beyond Binary Moral Judgment: Modeling Ethical Pluralism in AI [4.745265450673144]
  We propose a framework to model moral reasoning as a distribution over normative ethical theories or ethical pluralism.<n>A benchmark of 450 cases across 15 fine-grained subtheories was also prepared for the purposes of stacked ensemble learning.<n>Our experiments demonstrate that the integration of contextual and normative priors with the semantic embeddings significantly improves the performance of the classification.
  arXiv  Detail & Related papers  (2026-05-27T16:33:06Z)
• The Morality of Probability: How Implicit Moral Biases in LLMs May Shape the Future of Human-AI Symbiosis [33.50773360893016]
  This paper investigates how leading AI systems prioritize moral outcomes.<n>It shows that Care and Virtue values outcomes were rated most moral, while libertarian choices were consistently penalized.<n>It also highlights the need for explainability and cultural awareness as critical design principles to guide AI toward a transparent, aligned, and future symbiotic.
  arXiv  Detail & Related papers  (2025-09-12T14:37:57Z)
• "Pull or Not to Pull?'': Investigating Moral Biases in Leading Large Language Models Across Ethical Dilemmas [11.229443362516207]
  This study presents a comprehensive empirical evaluation of 14 leading large language models (LLMs)<n>We elicited 3,780 binary decisions and natural language justifications, enabling analysis along axes of decisional assertiveness, explanation answer consistency, public moral alignment, and sensitivity to ethically irrelevant cues.<n>We advocate for moral reasoning to become a primary axis in LLM alignment, calling for standardized benchmarks that evaluate not just what LLMs decide, but how and why.
  arXiv  Detail & Related papers  (2025-08-10T10:45:16Z)
• Moral Sparks in Social Media Narratives [14.025768295979184]
  We examine interactions on social media to understand human moral judgments in real-life ethical scenarios. Specifically, we examine posts from a popular Reddit subreddit (i.e., a subcommunity) called r/AmITheAsshole.
  arXiv  Detail & Related papers  (2023-10-30T05:03:26Z)
• What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations [48.686872351114964]
  Moral or ethical judgments rely heavily on the specific contexts in which they occur. We introduce defeasible moral reasoning: a task to provide grounded contexts that make an action more or less morally acceptable. We distill a high-quality dataset of 1.2M entries of contextualizations and rationales for 115K defeasible moral actions.
  arXiv  Detail & Related papers  (2023-10-24T00:51:29Z)
• Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories? [78.3738172874685]
  Making moral judgments is an essential step toward developing ethical AI systems. Prevalent approaches are mostly implemented in a bottom-up manner, which uses a large set of annotated data to train models based on crowd-sourced opinions about morality. This work proposes a flexible top-down framework to steer (Large) Language Models (LMs) to perform moral reasoning with well-established moral theories from interdisciplinary research.
  arXiv  Detail & Related papers  (2023-08-29T15:57:32Z)
• ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations [81.70195684646681]
  We present ClarifyDelphi, an interactive system that learns to ask clarification questions. We posit that questions whose potential answers lead to diverging moral judgments are the most informative. Our work is ultimately inspired by studies in cognitive science that have investigated the flexibility in moral cognition.
  arXiv  Detail & Related papers  (2022-12-20T16:33:09Z)
• Metaethical Perspectives on 'Benchmarking' AI Ethics [81.65697003067841]
  Benchmarks are seen as the cornerstone for measuring technical progress in Artificial Intelligence (AI) research. An increasingly prominent research area in AI is ethics, which currently has no set of benchmarks nor commonly accepted way for measuring the 'ethicality' of an AI system. We argue that it makes more sense to talk about 'values' rather than 'ethics' when considering the possible actions of present and future AI systems.
  arXiv  Detail & Related papers  (2022-04-11T14:36:39Z)
• Delphi: Towards Machine Ethics and Norms [38.8316885346292]
  We identify four underlying challenges towards machine ethics and norms. Our prototype model, Delphi, demonstrates strong promise of language-based commonsense moral reasoning. We present Commonsense Norm Bank, a moral textbook customized for machines.
  arXiv  Detail & Related papers  (2021-10-14T17:38:12Z)
• Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes [72.64975113835018]
  Motivated by descriptive ethics, we investigate a novel, data-driven approach to machine ethics. We introduce Scruples, the first large-scale dataset with 625,000 ethical judgments over 32,000 real-life anecdotes. Our dataset presents a major challenge to state-of-the-art neural language models, leaving significant room for improvement.
  arXiv  Detail & Related papers  (2020-08-20T17:34:15Z)
• Aligning AI With Shared Human Values [85.2824609130584]
  We introduce the ETHICS dataset, a new benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality. We find that current language models have a promising but incomplete ability to predict basic human ethical judgements. Our work shows that progress can be made on machine ethics today, and it provides a steppingstone toward AI that is aligned with human values.
  arXiv  Detail & Related papers  (2020-08-05T17:59:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.