World Properties without World Models: Recovering Spatial and Temporal Structure from Co-occurrence Statistics in Static Word Embeddings

• URL: http://arxiv.org/abs/2603.04317v1
• Date: Wed, 04 Mar 2026 17:37:05 GMT
• Title: World Properties without World Models: Recovering Spatial and Temporal Structure from Co-occurrence Statistics in Static Word Embeddings
• Abstract summary: We find substantial recoverable geographic signal and weaker but reliable temporal signal, with held-out R2 values of 0.71-0.87 for city coordinates and 0.48-0.52 for historical birth years.<n>These findings suggest that ordinary word co-occurrence preserves richer spatial, temporal, and environmental structure than is often assumed.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work interprets the linear recoverability of geographic and temporal variables from large language model (LLM) hidden states as evidence for world-like internal representations. We test a simpler possibility: that much of the relevant structure is already latent in text itself. Applying the same class of ridge regression probes to static co-occurrence-based embeddings (GloVe and Word2Vec), we find substantial recoverable geographic signal and weaker but reliable temporal signal, with held-out R^2 values of 0.71-0.87 for city coordinates and 0.48-0.52 for historical birth years. Semantic-neighbor analyses and targeted subspace ablations show that these signals depend strongly on interpretable lexical gradients, especially country names and climate-related vocabulary. These findings suggest that ordinary word co-occurrence preserves richer spatial, temporal, and environmental structure than is often assumed, revealing a remarkable and underappreciated capacity of simple static embeddings to preserve world-shaped structure from text alone. Linear probe recoverability alone therefore does not establish a representational move beyond text.

Related papers

• Understanding Temporal Semantic Stability in Open-Vocabulary UAV Perception through Metric 3D Fusion [0.0]
  We investigate temporal semantic stability by associating frame-wise predictions with persistent world-space locations.<n>Experiments on UAVid-3D reveal substantial frame-wise semantic flicker and show that high aggregate world-space agreement can overstate temporal stability.
  arXiv  Detail & Related papers  (2026-08-23T18:33:58Z)
• CoST: Semantic-Aware Urban Understanding via Spatial-Temporal Alignment [33.040248910330824]
  CoST aims to align spatial context with multi-temporal semantics to extract universal geographic regularities across regions.<n>Experiments demonstrate that CoST consistently achieves superior performance across various downstream tasks and in unseen scenario.
  arXiv  Detail & Related papers  (2026-08-21T12:36:13Z)
• Orthogonal JEPA: Factorized Predictive States for Latent World Models [95.20158869896393]
  We introduce method, a latent world-modeling framework based on predictive factorization.<n>We show that method can allocate redundant capacity to dominant signals while providing weak or conflicting gradients to less dominant predictive structure.<n>Experiments on controlled vision, single-cell transcriptomics, longitudinal health records, continuous control, and molecular dynamics evaluate representation quality, forecasting, planning, and long-horizon stability.
  arXiv  Detail & Related papers  (2026-08-20T13:59:57Z)
• SASGeo: Stability-Aware Semantic Map Localization for GNSS-Denied UAVs -- A Framework and Synthetic Proof of Concept [0.0]
  sas is a semantic map-localization framework that represents the environment through persistent structures.<n>sas combines semantic alignment, graph evidence, feature stability and geographic distinctiveness, explicit positive/contradictory/unknown observations, and integrity-aware rejection of ambiguous fixes.<n>Preliminary experiment does not validate real-flight navigation; rather, it demonstrates that structured semantic geometry can discriminate locations under controlled cross-view perturbations.
  arXiv  Detail & Related papers  (2026-07-07T16:57:43Z)
• Beyond Time Series: Spatial Reasoning for Epidemic Forecasting via Multimodal Learning [50.948271066372826]
  We introduce a spatially structured multimodal epidemic forecasting setting that integrates region-level temporal surveillance data with spatially localized auxiliary signals.<n>We evaluate our approach on real-world COVID-19, influenza, and influenza-like illness (IIL) forecasting tasks under realistic real-time evaluation protocols.
  arXiv  Detail & Related papers  (2026-06-20T17:58:17Z)
• Not All Points Are Equal: Uncertainty-Aware 4D LiDAR Scene Synthesis [63.07333625429548]
  We present U4D, a new framework that explicitly leverages spatial uncertainty to guide LiDAR scene generation.<n>Experiments on nuScenes and Semantic KITTI demonstrate state-of-the-art scene fidelity, temporal consistency, and downstream performance.
  arXiv  Detail & Related papers  (2026-06-01T17:24:14Z)
• Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory [54.947805187562274]
  We introduce realistic dialogues across diverse interaction scenarios that exhibit dynamic temporal evolution and long-term coherence.<n>The resulting benchmark encompasses challenging question-answer pairs spanning seven inquiry types.<n>We identify at least one of 27 critical memory characteristics that we identify as essential yet underexplored in current research.
  arXiv  Detail & Related papers  (2026-05-29T09:51:22Z)
• GeoViSTA: Geospatial Vision-Tabular Transformer for Multimodal Environment Representation [24.186635502118552]
  We propose GeoViSTA (Geospatial Vision-Tabular Transformer), a vision-tabular architecture that learns unified geospatial embeddings.<n>We train GeoViSTA with a self-supervised joint masked-autoencoding objective, forcing it to recover missing image patches.<n>Results demonstrate that jointly modeling the physical environment alongside structured socioeconomic context yields highly transferable representations.
  arXiv  Detail & Related papers  (2026-05-14T05:46:07Z)
• Long-Term Multi-Session 3D Reconstruction Under Substantial Appearance Change [52.46888249268445]
  Long-term environmental monitoring requires the ability to reconstruct and align 3D models across repeated site visits separated by months or years.<n>Existing approaches rely on post-hoc alignment of independently reconstructed sessions.<n>We propose enforcing cross-session correspondences directly within a joint SfM reconstruction.
  arXiv  Detail & Related papers  (2026-02-24T06:12:51Z)
• The Statistical Signature of LLMs [1.3135750017147134]
  We show that a simple, model-agnostic measure of statistical regularity differentiates generative regimes directly from surface text.<n>Across settings, compression reveals a persistent structural signature of probabilistic generation.<n>Our findings introduce a simple and robust framework for quantifying how generative systems reshape textual production.
  arXiv  Detail & Related papers  (2026-02-20T11:33:37Z)
• Symmetry in language statistics shapes the geometry of model representations [8.371687575566714]
  We show that the statistics of language exhibit a translation symmetry.<n>We prove that the latter governs the aforementioned geometric structures in high-dimensional word embedding models.<n>We show that this robustness naturally emerges if the co-occurrence statistics are collectively controlled by an underlying continuous latent variable.
  arXiv  Detail & Related papers  (2026-02-16T18:59:55Z)
• Hierarchical Geometry of Cognitive States in Transformer Embedding Spaces [0.0]
  We investigate whether sentence embeddings encode a graded, hierarchical structure aligned with human-interpretable cognitive or psychological attributes.<n>We construct a dataset of 480 natural-language sentences annotated with continuous ordinal energy scores and discrete tier labels.<n>Across models, both continuous scores and tier labels are reliably decodable, with shallow nonlinear probes providing consistent performance gains.
  arXiv  Detail & Related papers  (2025-12-23T03:37:34Z)
• One Swallow Does Not Make a Summer: Understanding Semantic Structures in Embedding Spaces [17.173074024116477]
  Embedding spaces are fundamental to modern AI, translating raw data into high-dimensional vectors that encode rich semantic relationships.<n>We introduce the Semantic Field Subspace (SFS), a geometry-preserving, context-aware representation that captures local semantic neighborhoods within the embedding space.<n>We also propose SAFARI, an unsupervised, modality-agnostic algorithm that uncovers hierarchical semantic structures using a novel metric called Semantic Shift.
  arXiv  Detail & Related papers  (2025-11-30T11:48:00Z)
• GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs [59.61242815508687]
  Graph neural networks (GNNs) on text--attributed graphs (TAGs) encode node texts using pretrained language models (PLMs) and propagate these embeddings through linear neighborhood aggregation.<n>This work introduces a local PCA-based metric that measures the degree of semantic drift and provides the first quantitative framework to analyze how different aggregation mechanisms affect manifold structure.
  arXiv  Detail & Related papers  (2025-11-12T06:48:43Z)
• Priors in Time: Missing Inductive Biases for Language Model Interpretability [58.07412640266836]
  We show that Sparse Autoencoders impose priors that assume independence of concepts across time, implying stationarity.<n>We introduce a new interpretability objective -- Temporal Feature Analysis -- which possesses a temporal inductive bias to decompose representations at a given time into two parts.<n>Our results underscore the need for inductive biases that match the data in designing robust interpretability tools.
  arXiv  Detail & Related papers  (2025-11-03T18:43:48Z)
• S$^2$Transformer: Scalable Structured Transformers for Global Station Weather Forecasting [67.93713728260646]
  Existing time series forecasting methods often ignore or unidirectionally model spatial correlation when conducting large-scale global station forecasting.<n>This contradicts the nature underlying observations of the global weather system limiting forecast performance.<n>We propose a novel Structured Spatial Attention in this paper.<n>It partitions the spatial graph into a set of subgraphs and instantiates Intra-subgraph Attention to learn local spatial correlation within each subgraph.<n>It aggregates nodes into subgraph representations for message passing among the subgraphs via Inter-subgraph Attention -- considering both spatial proximity and global correlation.
  arXiv  Detail & Related papers  (2025-09-10T05:33:28Z)
• Dense SAE Latents Are Features, Not Bugs [86.50389855919292]
  We show that dense latents serve functional roles in language model computation.<n>We identify classes tied to position tracking, context binding, entropy regulation, letter-specific output signals, part-of-speech, and principal component reconstruction.
  arXiv  Detail & Related papers  (2025-06-18T17:59:35Z)
• Typhoon Intensity Prediction with Vision Transformer [51.84456610977905]
  We introduce "Typhoon Intensity Transformer" (Tint) to predict typhoon intensity accurately across space and time. Tint uses self-attention mechanisms with global receptive fields per layer. Experiments on a publicly available typhoon benchmark validate the efficacy of Tint.
  arXiv  Detail & Related papers  (2023-11-28T03:11:33Z)
• Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching [60.645802236700035]
  Navigating drones through natural language commands remains challenging due to the dearth of accessible multi-modal datasets. We introduce GeoText-1652, a new natural language-guided geo-localization benchmark. This dataset is systematically constructed through an interactive human-computer process.
  arXiv  Detail & Related papers  (2023-11-21T17:52:30Z)
• $\text{H}^2\text{TNE}$: Temporal Heterogeneous Information Network Embedding in Hyperbolic Spaces [16.31067633778912]
  We propose a hyperbolic heterogeneous temporal network embedding model for temporal HINs. Specifically, we leverage a temporally and heterogeneously double-constrained random walk strategy to capture the structural and semantic information. Experimental results show that our method has superior performance on temporal link prediction and node classification compared with SOTA models.
  arXiv  Detail & Related papers  (2023-04-14T07:39:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.