Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension

• URL: http://arxiv.org/abs/2608.10296v1
• Date: Mon, 10 Aug 2026 23:03:32 GMT
• Title: Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension
• Abstract summary: We show that architectural variations within the dense transformer paradigm have a limited effect on accuracy.<n>We demonstrate that this is not the case in the long context setting.
• Score: 96.85790389598556
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One might imagine that architectural variations within the dense transformer paradigm have a limited effect on accuracy. However, we demonstrate that this is not the case in the long context setting. Specifically, we show that a set of four minor architectural decisions --- all made by at least one of the Olmo, Llama, and Qwen dense model families --- have a compoundingly negative effect on long context extensibility. Any one of these choices alone has a minor impact on long context performance, but combining three or more can drop the performance downstream by up to 47%. Furthermore, these differences are not detectable from short-context loss or validation datasets. We show that much of the variation in long context ability across model families is driven by these architectural features and detectable from applying context extension early in pretraining. We demonstrate this with controlled ablations that hold data, tokenizer, and extension recipe fixed while varying normalization, GQA, pretraining context length, and sliding window attention. After over 170,000 GPU hours of training, we release the resulting set of models as OlmPool, a set of 26 comparable 7B models with checkpoints before and after long-context extension. This pool includes several architectures that outperform the Llama 3 architecture on long context extensibility. In an analysis of our ablation models, we identify patterns in attention sink behavior and attention distributions across context that are attributable to specific architectural differences.

Related papers

• The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context [76.36635039699112]
  Large language models (LLMs) are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context.<n>We find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level.<n>This aggregate stability, however, masks significant per-example instability.
  arXiv  Detail & Related papers  (2026-07-14T17:01:12Z)
• DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction [21.818459524136376]
  DeepGaze3.5-VL establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of Information Gain (IG) on MIT1003, a 46% improvement over DeepGaze III.<n>Our generative framework serves as a powerful computational tool for direct behavioral interventions, allowing for controlled in-silico simulations that would be experimentally difficult or impossible to conduct in vivo.
  arXiv  Detail & Related papers  (2026-07-02T12:25:45Z)
• DARTS-GT: Differentiable Architecture Search for Graph Transformers with Quantifiable Instance-Specific Interpretability Analysis [0.0]
  Graph Transformers (GTs) have emerged as powerful architectures for graph-structured data, yet remain constrained by rigid designs and lack quantifiable interpretability.<n>We develop the first quantitative interpretability framework for GTs through causal ablation.<n>Experiments show DARTS-GT achieves state-of-the-art redesigns on four datasets while remaining competitive on others.<n>Our interpretability analysis reveals that visual attention salience and causal importance do not always correlate, indicating widely used visualization approaches may miss components that actually matter.
  arXiv  Detail & Related papers  (2025-10-16T06:15:42Z)
• Towards Long Context Hallucination Detection [49.195854802543714]
  Large Language Models (LLMs) have demonstrated remarkable performance across various tasks.<n>They are prone to contextual hallucination, generating information that is either unsubstantiated or contradictory to the given context.<n>We propose a novel architecture that enables pre-trained encoder models, such as BERT, to process long contexts and effectively detect contextual hallucinations.
  arXiv  Detail & Related papers  (2025-04-28T03:47:05Z)
• Is Temporal Prompting All We Need For Limited Labeled Action Recognition? [11.47868206641396]
  We present TP-CLIP, an adaptation of CLIP that leverages temporal visual prompting for temporal adaptation without modifying the core CLIP architecture.<n>TP-CLIP efficiently integrates into the CLIP architecture, leveraging its pre-trained capabilities for video data.
  arXiv  Detail & Related papers  (2025-04-02T16:50:28Z)
• In-Context Learning with Long-Context Models: An In-Depth Exploration [92.16922648612807]
  We show that, for many datasets with large label spaces, performance continues to increase with thousands of demonstrations.<n>We show that long-context ICL can be an effective tool, and may not require long-context for encoding the demonstration set at all.
  arXiv  Detail & Related papers  (2024-04-30T21:06:52Z)
• Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability [39.42414275888214]
  We evaluate 13 models capable of causal language modeling across a suite of synthetic in-context learning tasks. All the considered architectures can perform in-context learning under a wider range of conditions than previously documented. Several attention alternatives are sometimes competitive with or better in-context learners than transformers.
  arXiv  Detail & Related papers  (2023-10-12T05:43:06Z)
• Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts [65.84370471189676]
  We look at large-scale intermediate pre-training of decomposition-based transformers using distant supervision from comparable texts. We show that with such intermediate pre-training, developing robust decomposition-based models for a diverse range of tasks becomes more feasible.
  arXiv  Detail & Related papers  (2022-10-30T15:38:03Z)
• Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks [91.65637773358347]
  We propose a general graph neural network framework designed specifically for multivariate time series data. Our approach automatically extracts the uni-directed relations among variables through a graph learning module. Our proposed model outperforms the state-of-the-art baseline methods on 3 of 4 benchmark datasets.
  arXiv  Detail & Related papers  (2020-05-24T04:02:18Z)
• Segmentation of Macular Edema Datasets with Small Residual 3D U-Net Architectures [5.881334886616738]
  This paper investigates the application of deep convolutional neural networks with prohibitively small datasets to the problem of macular edema segmentation. We find that, contrary to popular belief, neural architectures within this application setting are able to achieve close to human-level performance on unseen test images without requiring large numbers of training examples.
  arXiv  Detail & Related papers  (2020-05-10T15:34:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.