Do BERT Embeddings Encode Narrative Dimensions? A Token-Level Probing Analysis of Time, Space, Causality, and Character in Fiction

• URL: http://arxiv.org/abs/2604.10786v1
• Date: Sun, 12 Apr 2026 19:23:48 GMT
• Title: Do BERT Embeddings Encode Narrative Dimensions? A Token-Level Probing Analysis of Time, Space, Causality, and Character in Fiction
• Abstract summary: This study investigates whether BERT embeddings encode dimensions of fictional narrative semantics.<n>A linear probe on BERT embeddings significantly outperforms a control probe on variance-matched random embeddings.
• Score: 2.9769545727805813
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Narrative understanding requires multidimensional semantic structures. This study investigates whether BERT embeddings encode dimensions of fictional narrative semantics -- time, space, causality, and character. Using an LLM to accelerate annotation, we construct a token-level dataset labeled with these four narrative categories plus "others." A linear probe on BERT embeddings (94% accuracy) significantly outperforms a control probe on variance-matched random embeddings (47%), confirming that BERT encodes meaningful narrative information. With balanced class weighting, the probe achieves a macro-average recall of 0.83, with moderate success on rare categories such as causality (recall = 0.75) and space (recall = 0.66). However, confusion matrix analysis reveals "Boundary Leakage," where rare dimensions are systematically misclassified as "others." Clustering analysis shows that unsupervised clustering aligns near-randomly with predefined categories (ARI = 0.081), suggesting that narrative dimensions are encoded but not as discretely separable clusters. Future work includes a POS-only baseline to disentangle syntactic patterns from narrative encoding, expanded datasets, and layer-wise probing.

Related papers

• GRACE: LLM-Grounded Semantic Metric Spaces for Scalable Mixed-Data Clustering [19.352298911016668]
  GRACE is a framework for scalable mixed-data clustering.<n>It maps heterogeneous values into knowledge-informed descriptions.<n>It achieves superior clustering accuracy and conceptual interpretability over 11 competing methods.
  arXiv  Detail & Related papers  (2026-08-08T03:14:50Z)
• Sparse Attention for Dense Open-Vocabulary Prediction in CLIP [54.45759517856271]
  We study an inference-time substitution of the row-wise softmax in the final visual self-attention layers.<n>Because entmax applies a data-dependent threshold that maps low scores exactly to zero, it acts as an implicit denoiser.<n>We evaluate on open-vocabulary tasks-dense semantic segmentation (Pascal VOC, Pascal Context, ADE20K) and fine-grained retrieval (FG-OVD)
  arXiv  Detail & Related papers  (2026-07-08T08:30:48Z)
• Bridging the Semantic Gap for Categorical Data Clustering via Large Language Models [64.58262227709842]
  ARISE (Attention-weighted Representation with Integrated Semantic Embeddings) is presented.<n>It builds semantic-aware representations that complement the metric space of categorical data for accurate clustering.<n>Experiments on eight benchmark datasets demonstrate consistent improvements over seven representative counterparts.
  arXiv  Detail & Related papers  (2026-01-03T11:37:46Z)
• Context-Aware Pseudo-Label Scoring for Zero-Shot Video Summarization [6.057968525653529]
  We propose a rubric-guided, pseudo-labeled, and prompt-driven zero-shot video summarization framework.<n>A small subset of human annotations is converted into high-confidence pseudo labels.<n>During inference, boundary scenes are scored independently based on their own descriptions.
  arXiv  Detail & Related papers  (2025-10-20T12:54:32Z)
• Topic Identification in LLM Input-Output Pairs through the Lens of Information Bottleneck [0.0]
  We develop a principled topic identification method grounded in the Deterministic Information Bottleneck (DIB) for geometric clustering.<n>Our key contribution is to transform the DIB method into a practical algorithm for high-dimensional data by substituting its intractable KL divergence term with a computationally efficient upper bound.
  arXiv  Detail & Related papers  (2025-08-26T20:00:51Z)
• Categorical Data Clustering via Value Order Estimated Distance Metric Learning [53.28598689867732]
  This paper introduces a novel order distance metric learning approach to intuitively represent categorical attribute values.<n>A new joint learning paradigm is developed to alternatively perform clustering and order distance metric learning.<n>The proposed method achieves superior clustering accuracy on categorical and mixed datasets.
  arXiv  Detail & Related papers  (2024-11-19T08:23:25Z)
• A Debiased Nearest Neighbors Framework for Multi-Label Text Classification [13.30576550077694]
  We introduce a DEbiased Nearest Neighbors (DENN) framework for Multi-Label Text Classification (MLTC) To address embedding alignment bias, we propose a debiased contrastive learning strategy, enhancing neighbor consistency on label co-occurrence. For confidence estimation bias, we present a debiased confidence estimation strategy, improving the adaptive combination of predictions from $k$NN and inductive binary classifications.
  arXiv  Detail & Related papers  (2024-08-06T14:00:23Z)
• A Fixed-Point Approach to Unified Prompt-Based Counting [51.20608895374113]
  This paper aims to establish a comprehensive prompt-based counting framework capable of generating density maps for objects indicated by various prompt types, such as box, point, and text. Our model excels in prominent class-agnostic datasets and exhibits superior performance in cross-dataset adaptation tasks.
  arXiv  Detail & Related papers  (2024-03-15T12:05:44Z)
• Understanding and Mitigating Spurious Correlations in Text Classification with Neighborhood Analysis [69.07674653828565]
  Machine learning models have a tendency to leverage spurious correlations that exist in the training set but may not hold true in general circumstances. In this paper, we examine the implications of spurious correlations through a novel perspective called neighborhood analysis. We propose a family of regularization methods, NFL (doN't Forget your Language) to mitigate spurious correlations in text classification.
  arXiv  Detail & Related papers  (2023-05-23T03:55:50Z)
• Intrinsic dimension estimation for discrete metrics [65.5438227932088]
  In this letter we introduce an algorithm to infer the intrinsic dimension (ID) of datasets embedded in discrete spaces. We demonstrate its accuracy on benchmark datasets, and we apply it to analyze a metagenomic dataset for species fingerprinting. This suggests that evolutive pressure acts on a low-dimensional manifold despite the high-dimensionality of sequences' space.
  arXiv  Detail & Related papers  (2022-07-20T06:38:36Z)
• Homonym Identification using BERT -- Using a Clustering Approach [0.0]
  This project aims to determine whether contextual information is sufficient for identifying a homonymous word. To capture the context, BERT embeddings are used as opposed to Word2Vec, which conflates senses into one vector. Various clustering algorithms are applied to the embeddings. Finally, the embeddings are visualized in a lower-dimensional space to understand the feasibility of the clustering process.
  arXiv  Detail & Related papers  (2021-01-07T06:26:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.