The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Era

• URL: http://arxiv.org/abs/2608.28980v2
• Date: Wed, 02 Sep 2026 04:57:21 GMT
• Title: The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Era
• Abstract summary: Review of 159 papers on machine learning across nine modalities.<n>No evidence of general architectural replacement for structured data models.<n>Although language-based models are highly competitive, whether the gap to a structure-aware architecture can eventually be eliminated remains untested.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Can the specialized architectures that machine learning has traditionally built for structured data be replaced by language-based models? This question is examined through a review of 159 papers (2016--2026) across nine modalities, with predictive accuracy considered alongside structural representation and computation. A distinction is made between performing a task and preserving and computing the structure that makes the task tractable, and existing approaches are organized into eight representational regimes, ranging from language-only systems to fully specialized architectures. Language-mediated models are found to be highly competitive in specific settings, including extreme few-shot prediction, discretized symbolic tasks, textually annotated knowledge graphs, and large-scale single-modality pretraining. However, whenever structural representation or computation is directly evaluated rather than accuracy alone, no evidence of general architectural replacement is found. Instead, a recurring pattern is observed across independent research communities: when language alone is insufficient, the missing structure is reintroduced through a graph module, structural tokens, specialized attention, or another non-linguistic component. In this sense, specialization more often relocates than disappears. Moreover, although performance of language-based models is improved by scaling, whether the gap to a structure-aware architecture can eventually be eliminated remains untested. The official repository for this work is available at https://github.com/kiyan-rezaee/language-vs-structure.

Related papers

• Graph Property Inference in Small Language Models: Effects of Representation and Inference Strategy [0.48302896549293584]
  It remains unclear how effectively limited-capacity language models can infer formal properties of relational structures when those structures are presented in textual form.<n>We conduct a systematic study of graph-theoretic property inference in small instruction-tuned language models.<n>We find that structural performance is highly sensitive to how relational information is organized.
  arXiv  Detail & Related papers  (2026-02-24T13:46:35Z)
• StructLens: A Structural Lens for Language Models via Maximum Spanning Trees [52.040177523973334]
  StructLens is an analytical framework designed to reveal how internal structures relate holistically.<n>Our findings demonstrate that StructLens yields an inter-layer similarity pattern that is distinctively different from conventional cosine similarity.
  arXiv  Detail & Related papers  (2026-02-10T11:30:32Z)
• Deep networks learn to parse uniform-depth context-free languages from local statistics [12.183764229746926]
  Understanding how the structure of language can be learned from sentences alone is a central question in both cognitive science and machine learning.<n>We introduce a class of context-free grammars (PCFGs) in which both the degree of ambiguity and the correlation structure across scales can be controlled.<n>We propose a unifying framework where correlations at different scales lift local ambiguities, enabling the emergence of hierarchical representations of the data.
  arXiv  Detail & Related papers  (2026-01-31T17:35:06Z)
• Structure-Aware Decoding Mechanisms for Complex Entity Extraction with Large-Scale Language Models [8.15127799301814]
  This paper proposes a structure-aware decoding method based on large language models.<n>It addresses the difficulty of maintaining both semantic integrity and structural consistency in nested and overlapping entity extraction tasks.<n> Experiments conducted on the ACE 2005 dataset demonstrate significant improvements in Accuracy, Precision, Recall, and F1-Score.
  arXiv  Detail & Related papers  (2025-12-16T00:40:06Z)
• Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures [49.19753720526998]
  We derive theoretical scaling laws for neural network performance on synthetic datasets.<n>We validate that convolutional networks, whose structure aligns with that of the generative process through locality and weight sharing, enjoy a faster scaling of performance.<n>This finding clarifies the architectural biases underlying neural scaling laws and highlights how representation learning is shaped by the interaction between model architecture and the statistical properties of data.
  arXiv  Detail & Related papers  (2025-05-11T17:44:14Z)
• Punctuation Restoration Improves Structure Understanding Without Supervision [5.925894224649895]
  We show that punctuation restoration as a learning objective improves performance on structure-related tasks.<n>Our results show that punctuation restoration is an effective learning objective that can improve structure understanding.
  arXiv  Detail & Related papers  (2024-02-13T11:22:52Z)
• SLOG: A Structural Generalization Benchmark for Semantic Parsing [68.19511282584304]
  The goal of compositional generalization benchmarks is to evaluate how well models generalize to new complex linguistic expressions. Existing benchmarks often focus on lexical generalization, the interpretation of novel lexical items in syntactic structures familiar from training, are often underrepresented. We introduce SLOG, a semantic parsing dataset that extends COGS with 17 structural generalization cases.
  arXiv  Detail & Related papers  (2023-10-23T15:39:09Z)
• Physics of Language Models: Part 1, Learning Hierarchical Language Structures [51.68385617116854]
  Transformer-based language models are effective but complex, and understanding their inner workings and reasoning mechanisms is a significant challenge.<n>We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences.<n>We demonstrate that generative models like GPT can accurately learn and reason over CFG-defined hierarchies and generate sentences based on it.
  arXiv  Detail & Related papers  (2023-05-23T04:28:16Z)
• Autoregressive Structured Prediction with Language Models [73.11519625765301]
  We describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at.
  arXiv  Detail & Related papers  (2022-10-26T13:27:26Z)
• A Machine Learning Approach to Classifying Construction Cost Documents into the International Construction Measurement Standard [0.0]
  We introduce the first automated models for classifying natural language descriptions provided in cost documents called "Bills of Quantities" We learn from a dataset of more than 50 thousand descriptions of items retrieved from 24 large infrastructure construction projects across the United Kingdom.
  arXiv  Detail & Related papers  (2022-10-24T11:35:53Z)
• Interpretable Mixture of Experts [71.55701784196253]
  Interpretable Mixture of Experts (IME) is an inherently-interpretable modeling framework. IME is demonstrated to be more accurate than single interpretable models and perform comparably with existing state-of-the-art Deep Neural Networks (DNNs) in accuracy. IME's explanations are compared to commonly-used post-hoc explanations methods through a user study.
  arXiv  Detail & Related papers  (2022-06-05T06:40:15Z)
• DeepStruct: Pretraining of Language Models for Structure Prediction [64.84144849119554]
  We pretrain language models on a collection of task-agnostic corpora to generate structures from text. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We show that a 10B parameter language model transfers non-trivially to most tasks and obtains state-of-the-art performance on 21 of 28 datasets.
  arXiv  Detail & Related papers  (2022-05-21T00:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.