Multi-Scale Probabilistic Generation Theory: A Hierarchical Framework for Interpreting Large Language Models

• URL: http://arxiv.org/abs/2505.18244v1
• Date: Fri, 23 May 2025 16:55:35 GMT
• Title: Multi-Scale Probabilistic Generation Theory: A Hierarchical Framework for Interpreting Large Language Models
• Authors: Yukin Zhang, Qi Dong,
• Abstract summary: Large Transformer based language models achieve remarkable performance but remain opaque in how they plan, structure, and realize text.<n>We introduce Multi_Scale Probabilistic Generation Theory (MSPGT), a hierarchical framework that factorizes generation into three semantic scales_global context, intermediate structure, and local word choices.
• Score: 1.2027959564488593
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Transformer based language models achieve remarkable performance but remain opaque in how they plan, structure, and realize text. We introduce Multi_Scale Probabilistic Generation Theory (MSPGT), a hierarchical framework that factorizes generation into three semantic scales_global context, intermediate structure, and local word choices and aligns each scale with specific layer ranges in Transformer architectures. To identify scale boundaries, we propose two complementary metrics: attention span thresholds and inter layer mutual information peaks. Across four representative models (GPT-2, BERT, RoBERTa, and T5), these metrics yield stable local/intermediate/global partitions, corroborated by probing tasks and causal interventions. We find that decoder_only models allocate more layers to intermediate and global processing while encoder_only models emphasize local feature extraction. Through targeted interventions, we demonstrate that local scale manipulations primarily influence lexical diversity, intermediate-scale modifications affect sentence structure and length, and global_scale perturbations impact discourse coherence all with statistically significant effects. MSPGT thus offers a unified, architecture-agnostic method for interpreting, diagnosing, and controlling large language models, bridging the gap between mechanistic interpretability and emergent capabilities.

Related papers

• Globalization for Scalable Short-term Load Forecasting [7.654516721062505]
  This paper investigates global load forecasting in the presence of data drifts.<n>We show how globalization, data heterogeneity, and data drift affect each differently.<n>We also examine the role of globalization in peak load forecasting and its potential for hierarchical forecasting.
  arXiv  Detail & Related papers  (2025-07-15T20:58:14Z)
• Multi-Scale Manifold Alignment: A Unified Framework for Enhanced Explainability of Large Language Models [4.084134914321567]
  Recent advances in Large Language Models (LLMs) have achieved strong performance, yet their internal reasoning remains opaque, limiting interpretability and trust in critical applications.<n>We propose a novel Multi_Scale Manifold Alignment framework that decomposes the latent space into global, intermediate, and local semantic Manifolds capturing themes, context, and word-level details.<n>This framework offers a unified explanation of how LLMs structure multi-scale semantics, advancing interpretability and enabling applications such as bias detection and robustness enhancement.
  arXiv  Detail & Related papers  (2025-05-24T10:25:58Z)
• Semantic Layered Embedding Diffusion in Large Language Models for Multi-Contextual Consistency [0.0]
  The Semantic Layered Embedding Diffusion (SLED) mechanism redefines the representation of hierarchical semantics within transformer-based architectures.<n>By introducing a multi-layered diffusion process grounded in spectral analysis, it achieves a complex balance between global and local semantic coherence.<n> Experimental results demonstrate significant improvements in perplexity and BLEU scores, emphasizing the mechanism's ability to adapt effectively across diverse domains.
  arXiv  Detail & Related papers  (2025-01-26T05:17:04Z)
• Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models [58.936893810674896]
  Face Anti-Spoofing (FAS) is essential for ensuring the security and reliability of facial recognition systems.<n>We introduce a multimodal large language model framework for FAS, termed Interpretable Face Anti-Spoofing (I-FAS)<n>We propose a Spoof-aware Captioning and Filtering (SCF) strategy to generate high-quality captions for FAS images.
  arXiv  Detail & Related papers  (2025-01-03T09:25:04Z)
• SJTU:Spatial judgments in multimodal models towards unified segmentation through coordinate detection [4.930667479611019]
  This paper introduces SJTU: Spatial Judgments in Multimodal Models - Towards Unified through Coordinate Detection.<n>It presents an approach for integrating segmentation techniques with vision-language models through spatial inference in multimodal space.<n>We demonstrate superior performance across benchmark datasets, achieving IoU scores of 0.5958 on COCO 2017 and 0.6758 on Pascal VOC.
  arXiv  Detail & Related papers  (2024-12-03T16:53:58Z)
• Interpreting token compositionality in LLMs: A robustness analysis [10.777646083061395]
  Constituent-Aware Pooling (CAP) is a methodology designed to analyse how large language models process linguistic structures.<n>CAP intervenes in model activations through constituent-based pooling at various model levels.<n>Our findings highlight fundamental limitations in current transformer architectures regarding compositional semantics processing and model interpretability.
  arXiv  Detail & Related papers  (2024-10-16T18:10:50Z)
• One-for-All: Towards Universal Domain Translation with a Single StyleGAN [86.33216867136639]
  We propose a novel translation model, UniTranslator, for transforming representations between visually distinct domains.<n>The proposed UniTranslator is versatile and capable of performing various tasks, including style mixing, stylization, and translations.<n>UniTranslator surpasses the performance of existing general-purpose models and performs well against specialized models in representative tasks.
  arXiv  Detail & Related papers  (2023-10-22T08:02:55Z)
• Investigating semantic subspaces of Transformer sentence embeddings through linear structural probing [2.5002227227256864]
  We present experiments with semantic structural probing, a method for studying sentence-level representations. We apply our method to language models from different families (encoder-only, decoder-only, encoder-decoder) and of different sizes in the context of two tasks. We find that model families differ substantially in their performance and layer dynamics, but that the results are largely model-size invariant.
  arXiv  Detail & Related papers  (2023-10-18T12:32:07Z)
• Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation [53.04781510348416]
  Video-based 3D human pose and shape estimations are evaluated by intra-frame accuracy and inter-frame smoothness. We propose to structurally decouple the modeling of long-term and short-term correlations in an end-to-end framework, Global-to-Local Transformer (GLoT) Our GLoT surpasses previous state-of-the-art methods with the lowest model parameters on popular benchmarks, i.e., 3DPW, MPI-INF-3DHP, and Human3.6M.
  arXiv  Detail & Related papers  (2023-03-26T14:57:49Z)
• Hierarchical Local-Global Transformer for Temporal Sentence Grounding [58.247592985849124]
  This paper studies the multimedia problem of temporal sentence grounding. It aims to accurately determine the specific video segment in an untrimmed video according to a given sentence query.
  arXiv  Detail & Related papers  (2022-08-31T14:16:56Z)
• A Variational Hierarchical Model for Neural Cross-Lingual Summarization [85.44969140204026]
  Cross-lingual summarization () is to convert a document in one language to a summary in another one. Existing studies on CLS mainly focus on utilizing pipeline methods or jointly training an end-to-end model. We propose a hierarchical model for the CLS task, based on the conditional variational auto-encoder.
  arXiv  Detail & Related papers  (2022-03-08T02:46:11Z)
• Examining Scaling and Transfer of Language Model Architectures for Machine Translation [51.69212730675345]
  Language models (LMs) process sequences in a single stack of layers, and encoder-decoder models (EncDec) utilize separate layer stacks for input and output processing. In machine translation, EncDec has long been the favoured approach, but with few studies investigating the performance of LMs.
  arXiv  Detail & Related papers  (2022-02-01T16:20:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.