NRGPT: An Energy-based Alternative for GPT

• URL: http://arxiv.org/abs/2512.16762v1
• Date: Thu, 18 Dec 2025 16:59:10 GMT
• Title: NRGPT: An Energy-based Alternative for GPT
• Abstract summary: Energy-based modeling is a different paradigm that views inference as a dynamical process operating on an energy landscape.<n>We propose a minimal modification of the GPT setting to unify it with the EBM framework.<n>We demonstrate that our model performs well for simple language (Shakespeare dataset), algebraic ListOPS tasks, and richer settings such as OpenWebText language modeling.
• Score: 14.661803594288024
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative Pre-trained Transformer (GPT) architectures are the most popular design for language modeling. Energy-based modeling is a different paradigm that views inference as a dynamical process operating on an energy landscape. We propose a minimal modification of the GPT setting to unify it with the EBM framework. The inference step of our model, which we call eNeRgy-GPT (NRGPT), is conceptualized as an exploration of the tokens on the energy landscape. We prove, and verify empirically, that under certain circumstances this exploration becomes gradient descent, although they don't necessarily lead to the best performing models. We demonstrate that our model performs well for simple language (Shakespeare dataset), algebraic ListOPS tasks, and richer settings such as OpenWebText language modeling. We also observe that our models may be more resistant to overfitting, doing so only during very long training.

Related papers

• The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model [0.0]
  We present a framework for modeling the energy consumption of Transformer training on multiple GPUs.<n>Inspired by roofline models, our approach incorporates a speedup-based hardware-efficiency factor.<n>We derive a scaling law model that accurately predicts training energy across heterogeneous configurations.
  arXiv  Detail & Related papers  (2026-06-22T16:18:06Z)
• HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents [71.09493646932046]
  We introduce HY-Embodied-0.5, a family of foundation models specifically designed for real-world embodied agents.<n>The HY-Embodied-0.5 suite comprises two primary variants: an efficient model with 2B activated parameters designed for edge deployment, and a powerful model with 32B activated parameters targeted for complex reasoning.<n>Our MoT-2B model outperforms similarly sized state-of-the-art models on 16 benchmarks, while the 32B variant achieves performance comparable to frontier models such as Gemini 3.0 Pro.
  arXiv  Detail & Related papers  (2026-04-08T17:59:48Z)
• Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector [0.0]
  We introduce EnergyGPT, a domain-specialized language model tailored for the energy sector.<n>We present a complete development pipeline, including data collection and curation.<n>We demonstrate that our training strategy enables improvements in domain relevance and performance without the need for large-scale infrastructure.
  arXiv  Detail & Related papers  (2025-09-08T19:48:52Z)
• Lumina-mGPT 2.0: Stand-Alone AutoRegressive Image Modeling [80.30976039119236]
  Lumina-mGPT 2.0 is a stand-alone, decoder-only autoregressive model.<n>It is trained entirely from scratch, enabling unrestricted architectural design and licensing freedom.<n>It achieves generation quality on par with state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2025-07-23T17:42:13Z)
• eP-ALM: Efficient Perceptual Augmentation of Language Models [70.47962271121389]
  We propose to direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning.
  arXiv  Detail & Related papers  (2023-03-20T19:20:34Z)
• Energy Transformer [64.22957136952725]
  Our work combines aspects of three promising paradigms in machine learning, namely, attention mechanism, energy-based models, and associative memory. We propose a novel architecture, called the Energy Transformer (or ET for short), that uses a sequence of attention layers that are purposely designed to minimize a specifically engineered energy function.
  arXiv  Detail & Related papers  (2023-02-14T18:51:22Z)
• Energy-Inspired Self-Supervised Pretraining for Vision Models [36.70550531181131]
  We introduce a self-supervised vision model pretraining framework inspired by energy-based models (EBMs) In the proposed framework, we model energy estimation and data restoration as the forward and backward passes of a single network. We show the proposed method delivers comparable and even better performance with remarkably fewer epochs of training.
  arXiv  Detail & Related papers  (2023-02-02T19:41:00Z)
• Versatile Energy-Based Probabilistic Models for High Energy Physics [0.0]
  We build a multi-purpose energy-based probabilistic model for High Energy Physics events at the Large Hadron Collider. This framework builds on a powerful generative model and describes higher-order inter-particle interactions.
  arXiv  Detail & Related papers  (2023-02-01T19:00:10Z)
• Re-parameterizing Your Optimizers rather than Architectures [119.08740698936633]
  We propose a novel paradigm of incorporating model-specific prior knowledge into Structurals and using them to train generic (simple) models. As an implementation, we propose a novel methodology to add prior knowledge by modifying the gradients according to a set of model-specific hyper- parameters. For a simple model trained with a Repr, we focus on a VGG-style plain model and showcase that such a simple model trained with a Repr, which is referred to as Rep-VGG, performs on par with the recent well-designed models.
  arXiv  Detail & Related papers  (2022-05-30T16:55:59Z)
• CoreLM: Coreference-aware Language Model Fine-Tuning [0.0]
  We propose a Fine-Tuning framework, named CoreLM, that extends the architecture of current Pretrained Language Models. We make available information outside the contextual space of the model, which results in a better Language Model for a fraction of the computational cost. Our proposed model achieves a lower Perplexity in GUMBY and LAMBDADA datasets when compared to GPT2 and a fine-tuned version of GPT2 without any changes.
  arXiv  Detail & Related papers  (2021-11-04T08:44:31Z)
• Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models [61.768082640087]
  We explore joint energy-based model (EBM) training during the finetuning of pretrained text encoders for natural language understanding tasks. Experiments show that EBM training can help the model reach a better calibration that is competitive to strong baselines.
  arXiv  Detail & Related papers  (2021-01-18T01:41:31Z)
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration [130.89746032163106]
  We propose ALOE, a new algorithm for learning conditional and unconditional EBMs for discrete structured data. We show that the energy function and sampler can be trained efficiently via a new variational form of power iteration. We present an energy model guided fuzzer for software testing that achieves comparable performance to well engineered fuzzing engines like libfuzzer.
  arXiv  Detail & Related papers  (2020-11-10T19:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.