How Language Models Organize and Structure Moral Knowledge

• URL: http://arxiv.org/abs/2608.27402v1
• Date: Thu, 27 Aug 2026 17:30:30 GMT
• Title: How Language Models Organize and Structure Moral Knowledge
• Abstract summary: We train six independent linear probes on open-weight language models.<n>We find directions neither collapse into a single moral detector nor isolate from one another.<n>The model represents moral tension itself, not a pre-resolved judgment.
• Score: 1.0829694003408499
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How do large language models (LLMs) organize moral knowledge? Models detect moral content broadly, but detection is a low bar. We ask whether they go further, distinguishing moral foundations from one another and organizing the relationships between them geometrically. We train six independent linear probes on open-weight language models, one per Moral Foundations Theory (MFT) category (care/harm, fair/cheat, lib/oppress, loy/betray, auth/subv, sanc/degrade), and examine how the resulting directions relate to each other in representation space. We find the directions neither collapse into a single moral detector nor isolate from one another. Rather, they span a near-maximal number of independent dimensions while sharing a positive common component. The shared component is the signature of integration, and it is moral-specific relative to a matched non-moral concept battery built identically (mean pairwise cosine 0.26 vs. 0.013). The geometry is consistent across architectures and scale and reaches its integration regime early in pre-training, well before probe accuracy saturates. The structure the model discovers shows no evidence of the individualizing/binding distinction predicted by Moral Foundations Theory (an underpowered test: only 20 candidate partitions exist) but rather reflects corpus statistics. Extending to moral dilemmas, each dilemma direction partially composes from its component foundations, at 2.7x a mismatched-pair baseline, while the majority of its variance encodes conflict-specific structure. The model represents moral tension itself, not a pre-resolved judgment.

Related papers

• Split the Labor: Separating Evidence Interpretation from Decision Aggregation [0.0]
  Systems that ask a language model to reach a conclusion from many sources usually conflate them into one prompt.<n>We propose a four-field evidence (hypothesis, reliability bucket, rationale, provenance) and show that fixing it determines both halves.<n>The fix is arithmetic rather than architectural, and applies to a class of rules beyond language models.
  arXiv  Detail & Related papers  (2026-08-14T17:24:55Z)
• Every Act Has Its Price: Compressed Moral Composition in Frontier LLMs [9.605269355662093]
  We introduce **Moral Trolley Arena**, a two-stage blind ELO benchmark for measuring how LLMs compose moral evidence.<n>The single-scene arena first calibrates individual moral acts from a 229-scenario corpus across five Moral Foundations Theory foundations.<n>The composite arena then combines calibrated acts into two-act moral items over a controlled intensity grid and measures the resulting composite preferences.
  arXiv  Detail & Related papers  (2026-05-29T02:36:10Z)
• Mechanistic Origin of Moral Indifference in Language Models [17.89411060814224]
  Existing behavioral alignment techniques for Large Language Models (LLMs) often neglect the discrepancy between surface compliance and internal unaligned representations.<n>We verify and remedy this indifference in LLMs' latent representations, utilizing 251k moral vectors constructed upon Prototype Theory and the Social-Chemistry-101 dataset.<n>We then employ Sparse Autoencoders on Qwen3-8B, isolate mono-semantic moral features, and targetedly reconstruct their topological relationships to align with ground-truth moral vectors.
  arXiv  Detail & Related papers  (2026-03-16T17:59:17Z)
• Tracing Moral Foundations in Large Language Models [33.17451937747417]
  We study how moral foundations are encoded, organized, and expressed within two instruction-tuned large language models.<n>We find that both models represent and distinguish moral foundations in a structured, layer-dependent way that aligns with human judgments.
  arXiv  Detail & Related papers  (2026-01-09T00:09:28Z)
• MORABLES: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fables [50.29407048003165]
  We present MORABLES, a human-verified benchmark built from fables and short stories drawn from historical literature.<n>The main task is structured as multiple-choice questions targeting moral inference, with carefully crafted distractors that challenge models to go beyond shallow, extractive question answering.<n>Our findings show that, while larger models outperform smaller ones, they remain susceptible to adversarial manipulation and often rely on superficial patterns rather than true moral reasoning.
  arXiv  Detail & Related papers  (2025-09-15T19:06:10Z)
• The Moral Mind(s) of Large Language Models [0.0]
  We show that large language models (LLMs) exhibit a consistent structure of moral preferences guiding their decisions.<n>Using a probabilistic rationality test, we found that at least one model from each major provider exhibited behavior consistent with approximately stable moral preferences.<n>We then estimated these utility functions and found that most models cluster around neutral moral stances.
  arXiv  Detail & Related papers  (2024-11-19T15:40:16Z)
• Exploring and steering the moral compass of Large Language Models [55.2480439325792]
  Large Language Models (LLMs) have become central to advancing automation and decision-making across various sectors. This study proposes a comprehensive comparative analysis of the most advanced LLMs to assess their moral profiles.
  arXiv  Detail & Related papers  (2024-05-27T16:49:22Z)
• What's "up" with vision-language models? Investigating their struggle with spatial reasoning [76.2406963762722]
  Three new corpora quantify model comprehension of basic spatial relations. We evaluate 18 vision-language (VL) models, finding that all perform poorly. We conclude by studying causes of this surprising behavior.
  arXiv  Detail & Related papers  (2023-10-30T17:50:15Z)
• What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations [48.686872351114964]
  Moral or ethical judgments rely heavily on the specific contexts in which they occur. We introduce defeasible moral reasoning: a task to provide grounded contexts that make an action more or less morally acceptable. We distill a high-quality dataset of 1.2M entries of contextualizations and rationales for 115K defeasible moral actions.
  arXiv  Detail & Related papers  (2023-10-24T00:51:29Z)
• Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories? [78.3738172874685]
  Making moral judgments is an essential step toward developing ethical AI systems. Prevalent approaches are mostly implemented in a bottom-up manner, which uses a large set of annotated data to train models based on crowd-sourced opinions about morality. This work proposes a flexible top-down framework to steer (Large) Language Models (LMs) to perform moral reasoning with well-established moral theories from interdisciplinary research.
  arXiv  Detail & Related papers  (2023-08-29T15:57:32Z)
• Pairwise Supervision Can Provably Elicit a Decision Boundary [84.58020117487898]
  Similarity learning is a problem to elicit useful representations by predicting the relationship between a pair of patterns. We show that similarity learning is capable of solving binary classification by directly eliciting a decision boundary.
  arXiv  Detail & Related papers  (2020-06-11T05:35:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.