Looking for the Inner Music: Probing LLMs' Understanding of Literary Style

• URL: http://arxiv.org/abs/2502.03647v1
• Date: Wed, 05 Feb 2025 22:20:17 GMT
• Title: Looking for the Inner Music: Probing LLMs' Understanding of Literary Style
• Authors: Rebecca M. M. Hicke, David Mimno,
• Abstract summary: Authorial style is easier to define than genre-level style. pronoun usage and word order prove significant for defining both kinds of literary style.
• Score: 3.5757761767474876
• License:
• Abstract: Recent work has demonstrated that language models can be trained to identify the author of much shorter literary passages than has been thought feasible for traditional stylometry. We replicate these results for authorship and extend them to a new dataset measuring novel genre. We find that LLMs are able to distinguish authorship and genre, but they do so in different ways. Some models seem to rely more on memorization, while others benefit more from training to learn author/genre characteristics. We then use three methods to probe one high-performing LLM for features that define style. These include direct syntactic ablations to input text as well as two methods that look at model internals. We find that authorial style is easier to define than genre-level style and is more impacted by minor syntactic decisions and contextual word usage. However, some traits like pronoun usage and word order prove significant for defining both kinds of literary style.

Related papers

• Beyond Profile: From Surface-Level Facts to Deep Persona Simulation in LLMs [50.0874045899661]
  We introduce CharacterBot, a model designed to replicate both the linguistic patterns and distinctive thought processes of a character. Using Lu Xun as a case study, we propose four training tasks derived from his 17 essay collections. These include a pre-training task focused on mastering external linguistic structures and knowledge, as well as three fine-tuning tasks. We evaluate CharacterBot on three tasks for linguistic accuracy and opinion comprehension, demonstrating that it significantly outperforms the baselines on our adapted metrics.
  arXiv  Detail & Related papers  (2025-02-18T16:11:54Z)
• A Distributional Perspective on Word Learning in Neural Language Models [57.41607944290822]
  There are no widely agreed-upon metrics for word learning in language models. We argue that distributional signatures studied in prior work fail to capture key distributional information. We obtain learning trajectories for a selection of small language models we train from scratch.
  arXiv  Detail & Related papers  (2025-02-09T13:15:59Z)
• Do LLMs write like humans? Variation in grammatical and rhetorical styles [0.7852714805965528]
  We study the rhetorical styles of large language models (LLMs) Using Douglas Biber's set of lexical, grammatical, and rhetorical features, we identify systematic differences between LLMs and humans. This demonstrates that despite their advanced abilities, LLMs struggle to match human styles.
  arXiv  Detail & Related papers  (2024-10-21T15:35:44Z)
• Capturing Style in Author and Document Representation [4.323709559692927]
  We propose a new architecture that learns embeddings for both authors and documents with a stylistic constraint. We evaluate our method on three datasets: a literary corpus extracted from the Gutenberg Project, the Blog Authorship and IMDb62.
  arXiv  Detail & Related papers  (2024-07-18T10:01:09Z)
• LFED: A Literary Fiction Evaluation Dataset for Large Language Models [58.85989777743013]
  We collect 95 literary fictions that are either originally written in Chinese or translated into Chinese, covering a wide range of topics across several centuries. We define a question taxonomy with 8 question categories to guide the creation of 1,304 questions. We conduct an in-depth analysis to ascertain how specific attributes of literary fictions (e.g., novel types, character numbers, the year of publication) impact LLM performance in evaluations.
  arXiv  Detail & Related papers  (2024-05-16T15:02:24Z)
• Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics [50.982315553104975]
  We investigate the bottom-up evolution of lexical semantics for a popular large language model, namely Llama2. Our experiments show that the representations in lower layers encode lexical semantics, while the higher layers, with weaker semantic induction, are responsible for prediction. This is in contrast to models with discriminative objectives, such as mask language modeling, where the higher layers obtain better lexical semantics.
  arXiv  Detail & Related papers  (2024-03-03T13:14:47Z)
• ParaGuide: Guided Diffusion Paraphrasers for Plug-and-Play Textual Style Transfer [57.6482608202409]
  Textual style transfer is the task of transforming stylistic properties of text while preserving meaning. We introduce a novel diffusion-based framework for general-purpose style transfer that can be flexibly adapted to arbitrary target styles. We validate the method on the Enron Email Corpus, with both human and automatic evaluations, and find that it outperforms strong baselines on formality, sentiment, and even authorship style transfer.
  arXiv  Detail & Related papers  (2023-08-29T17:36:02Z)
• Meta-Tuning LLMs to Leverage Lexical Knowledge for Generalizable Language Style Understanding [24.355564722047244]
  We show that current large language models struggle to capture some language styles without fine-tuning. We investigate whether LLMs can be meta-trained based on representative lexicons to recognize new styles they have not been fine-tuned on.
  arXiv  Detail & Related papers  (2023-05-24T00:17:36Z)
• PART: Pre-trained Authorship Representation Transformer [64.78260098263489]
  Authors writing documents imprint identifying information within their texts: vocabulary, registry, punctuation, misspellings, or even emoji usage. Previous works use hand-crafted features or classification tasks to train their authorship models, leading to poor performance on out-of-domain authors. We propose a contrastively trained model fit to learn textbfauthorship embeddings instead of semantics.
  arXiv  Detail & Related papers  (2022-09-30T11:08:39Z)
• Textual Stylistic Variation: Choices, Genres and Individuals [0.8057441774248633]
  This chapter argues for more informed target metrics for the statistical processing of stylistic variation in text collections. This chapter discusses variation given by genre, and contrasts it to variation occasioned by individual choice.
  arXiv  Detail & Related papers  (2022-05-01T16:39:49Z)
• From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey [10.822011920177408]
  Style transfer aims at re-writing existing texts and creating paraphrases that exhibit desired stylistic attributes. A handful of surveys give a methodological overview of the field, but they do not support researchers to focus on specific styles. We organize them into a hierarchy, highlighting the challenges for the definition of each of them, and pointing out gaps in the current research landscape.
  arXiv  Detail & Related papers  (2021-10-29T15:53:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.