Beyond Foundation Models: Dimension-Aware Neural Architecture Search with Small-Data Representation Models for Cryocooler Lifetime Prediction

• URL: http://arxiv.org/abs/2608.06993v1
• Date: Fri, 07 Aug 2026 09:11:45 GMT
• Title: Beyond Foundation Models: Dimension-Aware Neural Architecture Search with Small-Data Representation Models for Cryocooler Lifetime Prediction
• Abstract summary: We propose the FSD-RM (Family of Small-Data Representation Models) paradigm as a practical alternative for limited, domain-specific telemetry.<n>We focus on capacity-controlled representation learning using established encoder architectures.<n> Experiments on cryocooler telemetry show that the proposed approach achieves competitive predictive performance.
• Score: 3.40327344752292
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large-scale pretrained time-series models achieve strong results through large-scale pretraining and task-agnostic representation learning, but they rely on abundant, diverse data that industrial and scientific domains often lack. We therefore propose the FSD-RM (Family of Small-Data Representation Models) paradigm as a practical alternative for limited, domain-specific telemetry. Rather than relying on large-scale pretraining, we focus on capacity-controlled representation learning using established encoder architectures (CNN1D, LSTM, GRU, Transformer), selected for their suitability in small-data settings and interpretability. These encoders are trained unsupervised on multivariate telemetry data and integrated into a two-stage pipeline for downstream lifetime prediction. To systematically examine architectural trade-offs under data constraints, we employ \textbf{dimension-aware neural architecture search (NAS)} to jointly optimize model capacity and input dimensionality. Experiments on cryocooler telemetry show that the proposed approach achieves competitive predictive performance while reducing training cost and model complexity. The contribution lies in combining established representation learning techniques within a coherent, NAS-driven framework tailored to small-data regimes, with explicitly defined parameter settings and design choices. The results indicate that effective representation learning can be achieved without large-scale pretraining when appropriate inductive bias and capacity control are applied.

Related papers

• On Neural Scaling Laws for Weather Emulation through Continual Training [76.70310203875866]
  We study neural scaling in Scientific Machine Learning, focusing on models for weather forecasting.<n>To analyze scaling behavior in as simple a setting as possible, we adopt a minimal, scalable, general-purpose Swin Transformer architecture.<n>We show that models trained in this minimalist way follow predictable scaling trends and even outperform standard cosine learning rate schedules.
  arXiv  Detail & Related papers  (2026-03-26T17:37:25Z)
• LLM-Inspired Pretrain-Then-Finetune for Small-Data, Large-Scale Optimization [7.8639568562295965]
  We consider small-data, large-scale decision problems in which a firm must make many operational decisions simultaneously.<n>We propose a pretrain-then-finetune approach built on a designed Transformer model to address this challenge.
  arXiv  Detail & Related papers  (2026-02-03T16:08:33Z)
• Scaling Remote Sensing Foundation Models: Data Domain Tradeoffs at the Peta-Scale [0.0]
  We explore the scaling behaviors of artificial intelligence to establish techniques for training foundation models on high-resolution EO datasets.<n>We observe that even at this scale, performance is consistent with a data limited regime.<n>These practical insights are intended to inform data-collection strategies, compute budgets, and optimization schedules.
  arXiv  Detail & Related papers  (2025-12-29T23:53:11Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• Private Training & Data Generation by Clustering Embeddings [74.00687214400021]
  Differential privacy (DP) provides a robust framework for protecting individual data.<n>We introduce a novel principled method for DP synthetic image embedding generation.<n> Empirically, a simple two-layer neural network trained on synthetically generated embeddings achieves state-of-the-art (SOTA) classification accuracy.
  arXiv  Detail & Related papers  (2025-06-20T00:17:14Z)
• PMNO: A novel physics guided multi-step neural operator predictor for partial differential equations [23.04840527974364]
  We propose a novel physics guided multi-step neural operator (PMNO) architecture to address challenges in long-horizon prediction of complex physical systems.<n>The PMNO framework replaces the single-step input with multi-step historical data in the forward pass and introduces an implicit time-stepping scheme during backpropagation.<n>We demonstrate the superior predictive performance of PMNO predictor across a diverse range of physical systems.
  arXiv  Detail & Related papers  (2025-06-02T12:33:50Z)
• Remaining Useful Life Prediction: A Study on Multidimensional Industrial Signal Processing and Efficient Transfer Learning Based on Large Language Models [6.118896920507198]
  This paper introduces an innovative regression framework utilizing large language models (LLMs) for RUL prediction. Experiments on the Turbofan engine's RUL prediction task show that the proposed model surpasses state-of-the-art (SOTA) methods. With minimal target domain data for fine-tuning, the model outperforms SOTA methods trained on full target domain data.
  arXiv  Detail & Related papers  (2024-10-04T04:21:53Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Stochastic Configuration Machines for Industrial Artificial Intelligence [4.57421617811378]
  configuration networks (SCNs) play a key role in industrial artificial intelligence (IAI) This paper proposes a new randomized learner model, termed configuration machines (SCMs) to stress effective modelling and data size saving. Experimental studies are carried out over some benchmark datasets and three industrial applications.
  arXiv  Detail & Related papers  (2023-08-25T05:52:41Z)
• Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture [68.13678918660872]
  We design a more capable parameter-sharing architecture based on matrix product operator (MPO) MPO decomposition can reorganize and factorize the information of a parameter matrix into two parts. Our architecture shares the central tensor across all layers for reducing the model size.
  arXiv  Detail & Related papers  (2023-03-27T02:34:09Z)
• Rank-R FNN: A Tensor-Based Learning Model for High-Order Data Classification [69.26747803963907]
  Rank-R Feedforward Neural Network (FNN) is a tensor-based nonlinear learning model that imposes Canonical/Polyadic decomposition on its parameters. First, it handles inputs as multilinear arrays, bypassing the need for vectorization, and can thus fully exploit the structural information along every data dimension. We establish the universal approximation and learnability properties of Rank-R FNN, and we validate its performance on real-world hyperspectral datasets.
  arXiv  Detail & Related papers  (2021-04-11T16:37:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.