Causality $\neq$ Invariance: Function and Concept Vectors in LLMs

• URL: http://arxiv.org/abs/2602.22424v1
• Date: Wed, 25 Feb 2026 21:35:30 GMT
• Title: Causality $\neq$ Invariance: Function and Concept Vectors in LLMs
• Abstract summary: Do large language models (LLMs) represent concepts abstractly, i.e., independent of input format?<n>We revisit Function Vectors (FVs), compact representations of in-context learning (ICL) tasks that causally drive task performance.<n>We identify Concept Vectors (CVs), which carry more stable concept representations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Do large language models (LLMs) represent concepts abstractly, i.e., independent of input format? We revisit Function Vectors (FVs), compact representations of in-context learning (ICL) tasks that causally drive task performance. Across multiple LLMs, we show that FVs are not fully invariant: FVs are nearly orthogonal when extracted from different input formats (e.g., open-ended vs. multiple-choice), even if both target the same concept. We identify Concept Vectors (CVs), which carry more stable concept representations. Like FVs, CVs are composed of attention head outputs; however, unlike FVs, the constituent heads are selected using Representational Similarity Analysis (RSA) based on whether they encode concepts consistently across input formats. While these heads emerge in similar layers to FV-related heads, the two sets are largely distinct, suggesting different underlying mechanisms. Steering experiments reveal that FVs excel in-distribution, when extraction and application formats match (e.g., both open-ended in English), while CVs generalize better out-of-distribution across both question types (open-ended vs. multiple-choice) and languages. Our results show that LLMs do contain abstract concept representations, but these differ from those that drive ICL performance.

Related papers

• UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity [82.84279796262818]
  We propose UniFed-VLM, a unified instruction tuning framework for Vision-Language Models (VLMs)<n>It consists of two key components: 1) Federated Compensated Subspace Aggregation (FedCSA), which performs subspace-aligned aggregation of parameter-efficient adapters.<n>We conduct experiments on multiple benchmark datasets, and the results show that UniFed-VLM achieves stronger average performance.
  arXiv  Detail & Related papers  (2026-08-16T04:01:04Z)
• Fast & Faithful Function Vectors [24.484350585583233]
  Function vectors (FVs) are task representations elicited during in-context learning that can be used to steer Large Language Models (LLMs)<n>We study the impact of varying FV definitions for instructions along two degrees of freedom: attention head selection and steering.
  arXiv  Detail & Related papers  (2026-06-03T16:36:15Z)
• Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance [65.36257254806647]
  Vision-Language Models (VLMs) have enhanced traditional LLMs with visual capabilities through the integration of vision encoders.<n>We show that common practices, such as choosing encoders with the largest size or highest zero-shot accuracy, consistently fail to identify optimal models.<n>This intriguing finding begs a fundamental question: What factors of vision-encoders matter in VLM?
  arXiv  Detail & Related papers  (2026-05-02T08:42:13Z)
• Exploring Language-Agnosticity in Function Vectors: A Case Study in Machine Translation [18.767272825748456]
  Function vectors (FVs) are vector representations of tasks extracted from model activations during in-context learning.<n>We study whether FVs exhibit language-agnosticity, using machine translation as a case study.
  arXiv  Detail & Related papers  (2026-04-21T16:56:55Z)
• Rethinking Facial Expression Recognition in the Era of Multimodal Large Language Models: Benchmark, Datasets, and Beyond [116.65158801881984]
  We introduce post-training strategies aimed at enhancing the facial expression reasoning capabilities of MLLMs.<n>We develop a unified and interpretable FER foundation model termed UniFER-7B.
  arXiv  Detail & Related papers  (2025-11-01T03:53:00Z)
• Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models [19.361686225381447]
  Vision-language models (VLMs) are widely assumed to exhibit in-context learning (ICL)<n>We propose a new MM-ICL with Reasoning pipeline that augments each demonstration with a generated rationale alongside the answer.
  arXiv  Detail & Related papers  (2025-06-09T16:55:32Z)
• Analogical Reasoning Inside Large Language Models: Concept Vectors and the Limits of Abstraction [0.0]
  We show that function vectors (FVs) are not invariant to simple input changes, suggesting they capture more than pure concepts.<n>We localize a small set of attention heads that encode invariant concept vectors (CVs) for verbal concepts like "antonym"<n>CVs function as feature detectors that operate independently of the final output - meaning that a model may form a correct internal representation yet still produce an incorrect output.
  arXiv  Detail & Related papers  (2025-03-05T16:59:08Z)
• Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features [79.45405711339322]
  Generative Large Multimodal Models (LMMs) excel at a wide variety of vision-language (VL) tasks.<n>Despite strong performance, LMMs' generative outputs are not specialized for vision-language classification tasks.<n>We propose an approach that leverages multimodal feature extraction from the LMM's latent space.
  arXiv  Detail & Related papers  (2024-11-28T18:55:41Z)
• The representation landscape of few-shot learning and fine-tuning in large language models [43.76048699313088]
  In-context learning (ICL) and supervised fine-tuning (SFT) are two common strategies for improving the performance of modern large language models (LLMs) We analyze the probability landscape of their hidden representations in the two cases. We find that ICL and SFT create very different internal structures, in both cases undergoing a sharp transition in the middle of the network.
  arXiv  Detail & Related papers  (2024-09-05T16:15:12Z)
• Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs [63.29737699997859]
  Large Language Models (LLMs) have demonstrated impressive performance on multimodal tasks, without any multimodal finetuning. In this work, we expose frozen LLMs to image, video, audio and text inputs and analyse their internal representation.
  arXiv  Detail & Related papers  (2024-05-26T21:31:59Z)
• Function Vectors in Large Language Models [45.267194267587435]
  We report the presence of a simple neural mechanism that represents an input-output function as a vector within autoregressive transformer language models (LMs) Using causal mediation analysis on a diverse range of in-context-learning (ICL) tasks, we find that a small number attention heads transport a compact representation of the demonstrated task, which we call a function vector (FV)
  arXiv  Detail & Related papers  (2023-10-23T17:55:24Z)
• Delving into Multimodal Prompting for Fine-grained Visual Classification [57.12570556836394]
  Fine-grained visual classification (FGVC) involves categorizing fine subdivisions within a broader category. Recent advancements in pre-trained vision-language models have demonstrated remarkable performance in various high-level vision tasks. We propose a novel multimodal prompting solution, denoted as MP-FGVC, based on the contrastive language-image subcategory (CLIP) model.
  arXiv  Detail & Related papers  (2023-09-16T07:30:52Z)
• ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models [69.50316788263433]
  We propose ProbVLM, a probabilistic adapter that estimates probability distributions for the embeddings of pre-trained vision-language models. We quantify the calibration of embedding uncertainties in retrieval tasks and show that ProbVLM outperforms other methods. We present a novel technique for visualizing the embedding distributions using a large-scale pre-trained latent diffusion model.
  arXiv  Detail & Related papers  (2023-07-01T18:16:06Z)
• Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models [76.410400238974]
  We propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confident. A CLIP model is adopted as the reward model during TTA and provides feedback for the VLM. The proposed textitreinforcement learning with CLIP feedback(RLCF) framework is highly flexible and universal.
  arXiv  Detail & Related papers  (2023-05-29T11:03:59Z)
• CIPER: Combining Invariant and Equivariant Representations Using Contrastive and Predictive Learning [6.117084972237769]
  We introduce Contrastive Invariant and Predictive Equivariant Representation learning (CIPER) CIPER comprises both invariant and equivariant learning objectives using one shared encoder and two different output heads on top of the encoder. We evaluate our method on static image tasks and time-augmented image datasets.
  arXiv  Detail & Related papers  (2023-02-05T07:50:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.