Balancing Stylization and Truth via Disentangled Representation Steering

• URL: http://arxiv.org/abs/2508.04530v2
• Date: Thu, 07 Aug 2025 06:14:50 GMT
• Title: Balancing Stylization and Truth via Disentangled Representation Steering
• Abstract summary: A stylized large language model (LLM) responses via representation editing is a promising way for fine-grained output control.<n> imposing a distinctive style often degrades truthfulness.<n>Existing representation editing methods overlook this collateral impact and frequently contaminate the model's core truthfulness representations.<n>We propose StyliTruth, a mechanism that preserves stylization while keeping truthfulness intact.
• Score: 6.3136797036993375
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating stylized large language model (LLM) responses via representation editing is a promising way for fine-grained output control. However, there exists an inherent trade-off: imposing a distinctive style often degrades truthfulness. Existing representation editing methods, by naively injecting style signals, overlook this collateral impact and frequently contaminate the model's core truthfulness representations, resulting in reduced answer correctness. We term this phenomenon stylization-induced truthfulness collapse. We attribute this issue to latent coupling between style and truth directions in certain key attention heads, and propose StyliTruth, a mechanism that preserves stylization while keeping truthfulness intact. StyliTruth separates the style-relevant and truth-relevant subspaces in the model's representation space via an orthogonal deflation process. This decomposition enables independent control of style and truth in their own subspaces, minimizing interference. By designing adaptive, token-level steering vectors within each subspace, we dynamically and precisely control the generation process to maintain both stylistic fidelity and truthfulness. We validate our method on multiple styles and languages. Extensive experiments and analyses show that StyliTruth significantly reduces stylization-induced truthfulness collapse and outperforms existing inference-time intervention methods in balancing style adherence with truthfulness.

Related papers

• CV-DCLR: Causal-Visual Dynamic Label Refinement for Robust Zero-Shot Learning [25.21342780360303]
  We propose the Causal-Visual Dynamic Label Refinement framework.<n>It recalibrates visual-semantic associations via a Dual-Stream Mutual Correction Mechanism.<n>Experiments on the CUB, SUN, and AWA2 benchmarks demonstrate that CV-DCLR significantly outperforms state-of-the-art methods in high-ambiguity scenarios.
  arXiv  Detail & Related papers  (2026-07-01T12:45:35Z)
• LatRef-Diff: Latent and Reference-Guided Diffusion for Facial Attribute Editing and Style Manipulation [78.6161238980415]
  Conditional GANs have shown progress, but they are limited by accuracy issues and training instability.<n>We propose LatRef-Diff, a novel diffusion-based framework that addresses these limitations.<n>We show that LatRef-Diff achieves state-of-the-art performance in both qualitative and quantitative evaluations.
  arXiv  Detail & Related papers  (2026-04-23T04:47:40Z)
• Implicit Style Conditioning: A Structured Style-Rewrite Framework for Low-Resource Character Modeling [0.0]
  Small Language Models (SLMs) with highly stylized personas remains a challenge due to data scarcity and the complexity of style disentanglement.<n>We propose a Structured Style-Rewrite Framework that explicitly disentangles style into three interpretable dimensions.<n>By leveraging explicit reasoning traces during training as a strong inductive bias, our approach aligns the model's latent representations with structured style features.
  arXiv  Detail & Related papers  (2026-03-06T06:04:47Z)
• Controlling Chat Style in Language Models via Single-Direction Editing [43.11304710234668]
  This paper tests the hypothesis that distinct stylistic attributes are encoded as linear directions in the model's activation space.<n>We present a lightweight, training-free method for precise style control.<n>Our approach supports linear style composition, enhances safety by ablating undesirable behaviors, and, as confirmed by experiments on over a dozen models, achieves high style adherence while preserving core capabilities at minimal computational cost.
  arXiv  Detail & Related papers  (2026-02-10T06:05:04Z)
• DICE: Disentangling Artist Style from Content via Contrastive Subspace Decomposition in Diffusion Models [29.817934937899196]
  DICE is a training-free framework for on-the-fly artist style erasure.<n>We construct contrastive triplets to compel the model to distinguish between style and non-style features in the latent space.<n>Experiments demonstrate that DICE achieves a superior balance between the thoroughness of style erasure and the preservation of content integrity.
  arXiv  Detail & Related papers  (2026-02-08T17:06:48Z)
• MIRRORTALK: Forging Personalized Avatars Via Disentangled Style and Hierarchical Motion Control [42.87874090062771]
  MirrorTalk is a generative framework based on a conditional diffusion model.<n>It can distill pure style representations from a brief reference video.<n>It achieves significant improvements over state-of-the-art methods in terms of lip-sync accuracy and personalization preservation.
  arXiv  Detail & Related papers  (2026-01-30T03:23:01Z)
• Sissi: Zero-shot Style-guided Image Synthesis via Semantic-style Integration [57.02757226679549]
  We introduce a training-free framework that reformulates style-guided synthesis as an in-context learning task.<n>We propose a Dynamic Semantic-Style Integration (DSSI) mechanism that reweights attention between semantic and style visual tokens.<n>Experiments show that our approach achieves high-fidelity stylization with superior semantic-style balance and visual quality.
  arXiv  Detail & Related papers  (2026-01-10T16:01:14Z)
• Emergence of Linear Truth Encodings in Language Models [64.86571541830598]
  Large language models exhibit linear subspaces that separate true from false statements, yet the mechanism behind their emergence is unclear.<n>We introduce a transparent, one-layer transformer toy model that reproduces such truth subspaces end-to-end.<n>We study one simple setting in which truth encoding can emerge, encouraging the model to learn this distinction in order to lower the LM loss on future tokens.
  arXiv  Detail & Related papers  (2025-10-17T16:30:07Z)
• Disentangling Content from Style to Overcome Shortcut Learning: A Hybrid Generative-Discriminative Learning Framework [4.7403081236484335]
  Shortcut Learning exploits superficial features like texture instead of intrinsic structure.<n>We propose HyGDL, a hybrid framework that achieves explicit content disentanglement.<n>Unlike prior methods, this principled disentanglement allows HyGDL to learn truly robust representations.
  arXiv  Detail & Related papers  (2025-09-15T05:28:32Z)
• Large Language Models Encode Semantics in Low-Dimensional Linear Subspaces [31.401762286885656]
  Understanding the space geometry of large language models (LLMs) is key to interpreting their behavior and improving alignment.<n>baturayWe investigate what extent LLMs internally organize related to semantic understanding.
  arXiv  Detail & Related papers  (2025-07-13T17:03:25Z)
• Be Decisive: Noise-Induced Layouts for Multi-Subject Generation [56.80513553424086]
  Complex prompts lead to subject leakage, causing inaccuracies in quantities, attributes, and visual features.<n>We introduce a new approach that predicts a spatial layout aligned with the prompt, derived from the initial noise, and refines it throughout the denoising process.<n>Our method employs a small neural network to predict and refine the evolving noise-induced layout at each denoising step.
  arXiv  Detail & Related papers  (2025-05-27T17:54:24Z)
• Factual Self-Awareness in Language Models: Representation, Robustness, and Scaling [56.26834106704781]
  Factual incorrectness in generated content is one of the primary concerns in ubiquitous deployment of large language models (LLMs)<n>We provide evidence supporting the presence of LLMs' internal compass that dictate the correctness of factual recall at the time of generation.<n>Scaling experiments across model sizes and training dynamics highlight that self-awareness emerges rapidly during training and peaks in intermediate layers.
  arXiv  Detail & Related papers  (2025-05-27T16:24:02Z)
• DRESSing Up LLM: Efficient Stylized Question-Answering via Style Subspace Editing [23.467409274256255]
  DRESS is a novel approach for generating stylized large language model (LLM) responses through representation editing.<n>Our approach disentangles a style-relevant subspace within the model's representation space to conduct representation editing.<n>In short, DRESS is a lightweight, train-free solution for enhancing LLMs with flexible and effective style control.
  arXiv  Detail & Related papers  (2025-01-24T10:04:53Z)
• Z-STAR+: A Zero-shot Style Transfer Method via Adjusting Style Distribution [24.88532732093652]
  Style transfer presents a significant challenge, primarily centered on identifying an appropriate style representation.<n>In contrast to existing approaches, we have discovered that latent features in vanilla diffusion models inherently contain natural style and content distributions.<n>Our method adopts dual denoising paths to represent content and style references in latent space, subsequently guiding the content image denoising process with style latent codes.
  arXiv  Detail & Related papers  (2024-11-28T15:56:17Z)
• DiffArtist: Towards Structure and Appearance Controllable Image Stylization [19.5597806965592]
  We present a comprehensive study on the simultaneous stylization of structure and appearance of 2D images.<n>We introduce DiffArtist, which is the first stylization method to allow for dual controllability over structure and appearance.
  arXiv  Detail & Related papers  (2024-07-22T17:58:05Z)
• Deformable One-shot Face Stylization via DINO Semantic Guidance [12.771707124161665]
  This paper addresses the issue of one-shot face stylization, focusing on the simultaneous consideration of appearance and structure. We explore deformation-aware face stylization that diverges from traditional single-image style reference, opting for a real-style image pair instead.
  arXiv  Detail & Related papers  (2024-03-01T11:30:55Z)
• ALADIN-NST: Self-supervised disentangled representation learning of artistic style through Neural Style Transfer [60.6863849241972]
  We learn a representation of visual artistic style more strongly disentangled from the semantic content depicted in an image. We show that strongly addressing the disentanglement of style and content leads to large gains in style-specific metrics.
  arXiv  Detail & Related papers  (2023-04-12T10:33:18Z)
• Towards Counterfactual Image Manipulation via CLIP [106.94502632502194]
  Existing methods can achieve realistic editing of different visual attributes such as age and gender of facial images. We investigate this problem in a text-driven manner with Contrastive-Language-Image-Pretraining (CLIP) We design a novel contrastive loss that exploits predefined CLIP-space directions to guide the editing toward desired directions from different perspectives.
  arXiv  Detail & Related papers  (2022-07-06T17:02:25Z)
• Where and What? Examining Interpretable Disentangled Representations [96.32813624341833]
  Capturing interpretable variations has long been one of the goals in disentanglement learning. Unlike the independence assumption, interpretability has rarely been exploited to encourage disentanglement in the unsupervised setting. In this paper, we examine the interpretability of disentangled representations by investigating two questions: where to be interpreted and what to be interpreted.
  arXiv  Detail & Related papers  (2021-04-07T11:22:02Z)
• Improving Disentangled Text Representation Learning with Information-Theoretic Guidance [99.68851329919858]
  discrete nature of natural language makes disentangling of textual representations more challenging. Inspired by information theory, we propose a novel method that effectively manifests disentangled representations of text. Experiments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation.
  arXiv  Detail & Related papers  (2020-06-01T03:36:01Z)
• Learning to Manipulate Individual Objects in an Image [71.55005356240761]
  We describe a method to train a generative model with latent factors that are independent and localized. This means that perturbing the latent variables affects only local regions of the synthesized image, corresponding to objects. Unlike other unsupervised generative models, ours enables object-centric manipulation, without requiring object-level annotations.
  arXiv  Detail & Related papers  (2020-04-11T21:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.