Decomposing how prompting steers behavior

• URL: http://arxiv.org/abs/2606.03093v1
• Date: Tue, 02 Jun 2026 03:27:24 GMT
• Title: Decomposing how prompting steers behavior
• Abstract summary: We decompose prompt-induced representational change into interpretable geometric components.<n>Our framework reveals how models route task-relevant structure to produce prompt-driven behavior.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompting steers large language models (LLMs) and vision-language models (VLMs) without weight updates, but it remains unclear how instruction changes reshape internal representations to produce behavior. We introduce a nested geometric decomposition framework that treats prompting as a transformation of the representational geometry of the content following the prompt. For each prompt pair, we align representations of the same stimuli under two prompts using increasingly expressive stimulus-invariant maps: translation, rigid transformation with uniform scaling, sequential axis scaling, affine transformation, and nonlinear transformation. We then causally test each map by replacing a single layer's prompt-A hidden state for held-out stimuli with its mapped counterpart and measuring recovery of prompt-B representational geometry and behavior. Across three LLMs, three VLMs, and six text or image datasets spanning style, emotion, scene content, and number, prompts consistently reshape representations toward the instructed task structure. Cross-validated variance decomposition shows that much prompt-induced activation change is captured by shape-preserving maps, especially translation and rigid transformation with uniform scaling, while tier profiles reveal model- and task-specific routing strategies across layers. Crucially, although translation and rigid tiers already improve behavioral agreement, affine transformation is the first tier to nearly recover target-prompt task geometry and yields corresponding behavioral gains. This suggests that cross-dimensional linear mixing is a key mechanism by which prompts reorganize representations toward instructed task structure. Our framework decomposes prompt-induced representational change into interpretable geometric components and reveals how models route task-relevant structure to produce prompt-driven behavior.

Related papers

• The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers [6.995112644485604]
  We ask whether the grammatical role of tokens, as marked by their part-of-speech (PoS) tag, shapes the local geometry of this manifold.<n>We show that geometric features alone recover a token's grammatical role, and use them to interpret how the semantic content of each PoS evolves across layers in a downstream classification task.
  arXiv  Detail & Related papers  (2026-08-25T21:27:21Z)
• The Push-Forward Transform for Continuous and Robust Comparison of Dynamic Shapes [1.6058099298620423]
  We introduce a mathematical framework for shape comparison based on mapping functions from the shape domain to a common reference domain.<n>We derive an interpretable morphometric that quantifies shape similarity and reveals features such as skeletal topology and rotational symmetries.<n>We present the mathematical formulation, describe an efficient algorithm, and benchmark the approach on 2D, 3D, and temporal data sets.
  arXiv  Detail & Related papers  (2026-08-03T14:31:21Z)
• Video Analysis and Generation via a Semantic Progress Function [75.07700965101854]
  We introduce a one-dimensional representation that captures how the meaning of a given sequence evolves over time.<n>For each frame, we compute distances between semantic embeddings and fit a smooth curve that reflects the cumulative semantic shift across the sequence.<n>Building on this insight, we propose a semantic linearization procedure that re parameterizes the sequence so that semantic change unfolds at a constant rate.
  arXiv  Detail & Related papers  (2026-04-24T13:48:51Z)
• Learning 3D Representations for Spatial Intelligence from Unposed Multi-View Images [81.94999489820974]
  UniSplat is a feed-forward framework for learning 3D representations from unposed multi-view images.<n>We introduce a dual-masking strategy that strengthens geometry induction in the encoder.<n>Second, we develop a coarse-to-fine Gaussian splatting strategy that reduces appearance-semantics inconsistencies.<n>Third, we introduce a pose-conditioned recalibration mechanism that interrelates predicted 3D point and semantic maps into the image plane.
  arXiv  Detail & Related papers  (2026-04-12T10:36:18Z)
• Context Structure Reshapes the Representational Geometry of Language Models [9.670218260803628]
  Large Language Models (LLMs) organize the representations of input sequences into straighter neural trajectories.<n>Recent work has shown that this in-context learning can be reflected in representational changes.<n>We measure representational straightening in Gemma 2 models across a diverse set of in-context tasks.
  arXiv  Detail & Related papers  (2026-01-29T22:17:17Z)
• Talk2Move: Reinforcement Learning for Text-Instructed Object-Level Geometric Transformation in Scenes [69.4534914304302]
  We introduce Talk2Move, a framework for text-instructed spatial transformation of objects within scenes.<n>Talk2Move employs Group Relative Policy Optimization to explore geometric actions through diverse rollouts.<n> Experiments on curated benchmarks demonstrate that Talk2Move achieves precise, consistent, and semantically faithful object transformations.
  arXiv  Detail & Related papers  (2026-01-05T18:55:32Z)
• GeoSketch: A Neural-Symbolic Approach to Geometric Multimodal Reasoning with Auxiliary Line Construction and Affine Transformation [28.500787311066563]
  GeoSketch is a neural-symbolic framework that recasts geometric reasoning as an interactive perception-reasoning-action loop.<n>By unifying hierarchical decision-making, executable visual actions, and symbolic verification, GeoSketch advances multimodal reasoning from static interpretation to dynamic interaction.
  arXiv  Detail & Related papers  (2025-09-26T15:12:04Z)
• Large Spatial Model: End-to-end Unposed Images to Semantic 3D [79.94479633598102]
  Large Spatial Model (LSM) processes unposed RGB images directly into semantic radiance fields. LSM simultaneously estimates geometry, appearance, and semantics in a single feed-forward operation. It can generate versatile label maps by interacting with language at novel viewpoints.
  arXiv  Detail & Related papers  (2024-10-24T17:54:42Z)
• Learning SO(3)-Invariant Semantic Correspondence via Local Shape Transform [62.27337227010514]
  We introduce a novel self-supervised Rotation-Invariant 3D correspondence learner with Local Shape Transform, dubbed RIST. RIST learns to establish dense correspondences between shapes even under challenging intra-class variations and arbitrary orientations. RIST demonstrates state-of-the-art performances on 3D part label transfer and semantic keypoint transfer given arbitrarily rotated point cloud pairs.
  arXiv  Detail & Related papers  (2024-04-17T08:09:25Z)
• Frame Averaging for Equivariant Shape Space Learning [85.42901997467754]
  A natural way to incorporate symmetries in shape space learning is to ask that the mapping to the shape space (encoder) and mapping from the shape space (decoder) are equivariant to the relevant symmetries. We present a framework for incorporating equivariance in encoders and decoders by introducing two contributions.
  arXiv  Detail & Related papers  (2021-12-03T06:41:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.