Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector

• URL: http://arxiv.org/abs/2509.07177v1
• Date: Mon, 08 Sep 2025 19:48:52 GMT
• Title: Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector
• Abstract summary: We introduce EnergyGPT, a domain-specialized language model tailored for the energy sector.<n>We present a complete development pipeline, including data collection and curation.<n>We demonstrate that our training strategy enables improvements in domain relevance and performance without the need for large-scale infrastructure.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models have demonstrated impressive capabilities across various domains. However, their general-purpose nature often limits their effectiveness in specialized fields such as energy, where deep technical expertise and precise domain knowledge are essential. In this paper, we introduce EnergyGPT, a domain-specialized language model tailored for the energy sector, developed by fine-tuning LLaMA 3.1-8B model using Supervised Fine-Tuning on a high-quality, curated corpus of energy-related texts. We present a complete development pipeline, including data collection and curation, model fine-tuning, benchmark design and LLM-judge choice, evaluation and deployment. Through this work, we demonstrate that our training strategy enables improvements in domain relevance and performance without the need for large-scale infrastructure. By evaluating the performance of the model using domain-specific question-answering benchmarks, our results demonstrate that EnergyGPT outperforms the base model in most of the energy-related language understanding and generation tasks.

Related papers

• FETS Benchmark: Foundation Models Outperform Dataset-specific Machine Learning in Energy Time Series Forecasting [0.39373541926236766]
  Foundation models that aim to learn generalizable patterns via extensive pretraining have shown superior performance in multiple prediction tasks.<n>Despite their success and strong potential to address challenges in energy forecasting, their application in this domain remains largely unexplored.
  arXiv  Detail & Related papers  (2026-04-24T08:00:50Z)
• NRGPT: An Energy-based Alternative for GPT [14.661803594288024]
  Energy-based modeling is a different paradigm that views inference as a dynamical process operating on an energy landscape.<n>We propose a minimal modification of the GPT setting to unify it with the EBM framework.<n>We demonstrate that our model performs well for simple language (Shakespeare dataset), algebraic ListOPS tasks, and richer settings such as OpenWebText language modeling.
  arXiv  Detail & Related papers  (2025-12-18T16:59:10Z)
• From Prompts to Power: Measuring the Energy Footprint of LLM Inference [1.9116784879310027]
  We present a large-scale measurement-based study comprising over 32,500 measurements across 21 GPU configurations and 155 model architectures.<n>Using the vLLM inference engine, we quantify energy usage at the prompt level and identify how architectural and operational factors shape energy demand.<n>We develop a predictive model that accurately estimates inference energy consumption across unseen architectures and hardware, and implement it as a browser extension to raise awareness of the environmental impact of generative AI.
  arXiv  Detail & Related papers  (2025-11-05T15:06:46Z)
• Comparing energy consumption and accuracy in text classification inference [0.9208007322096533]
  This study systematically evaluates the trade-offs between model accuracy and energy consumption in text classification inference.<n>The best-performing model in terms of accuracy can also be energy-efficient, while larger LLMs tend to consume significantly more energy with lower classification accuracy.
  arXiv  Detail & Related papers  (2025-08-19T18:00:08Z)
• Energy Considerations of Large Language Model Inference and Efficiency Optimizations [28.55549828393871]
  As large language models (LLMs) scale in size and adoption, their computational and environmental costs continue to rise.<n>We systematically analyze the energy implications of common inference efficiency optimizations across diverse NLP and AI workloads.<n>Our findings reveal that the proper application of relevant inference efficiency optimizations can reduce total energy use by up to 73% from unoptimized baselines.
  arXiv  Detail & Related papers  (2025-04-24T15:45:05Z)
• Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models [20.730898779471556]
  Large language models (LLMs) are increasingly recognized for their exceptional generative capabilities and versatility.<n>This study conducts a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks.<n>We find that quantization and optimal batch sizes, along with targeted prompt phrases, can significantly reduce energy usage.
  arXiv  Detail & Related papers  (2025-02-08T15:34:52Z)
• Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian [75.94354349994576]
  This paper explores the feasibility of employing smaller, domain-specific encoder LMs alongside prompting techniques to enhance performance in specialized contexts. Our study concentrates on the Italian bureaucratic and legal language, experimenting with both general-purpose and further pre-trained encoder-only models. The results indicate that while further pre-trained models may show diminished robustness in general knowledge, they exhibit superior adaptability for domain-specific tasks, even in a zero-shot setting.
  arXiv  Detail & Related papers  (2024-07-30T08:50:16Z)
• WorldGPT: Empowering LLM as Multimodal World Model [51.243464216500975]
  We introduce WorldGPT, a generalist world model built upon Multimodal Large Language Model (MLLM) WorldGPT acquires an understanding of world dynamics through analyzing millions of videos across various domains. We conduct evaluations on WorldNet, a multimodal state transition prediction benchmark.
  arXiv  Detail & Related papers  (2024-04-28T14:42:02Z)
• Advancing Building Energy Modeling with Large Language Models: Exploration and Case Studies [2.8879609855863713]
  The rapid progression in artificial intelligence has facilitated the emergence of large language models like ChatGPT. This paper investigates the innovative integration of large language models with building energy modeling software.
  arXiv  Detail & Related papers  (2024-02-14T21:02:07Z)
• On Feature Diversity in Energy-based Models [98.78384185493624]
  An energy-based model (EBM) is typically formed of inner-model(s) that learn a combination of the different features to generate an energy mapping for each input configuration. We extend the probably approximately correct (PAC) theory of EBMs and analyze the effect of redundancy reduction on the performance of EBMs.
  arXiv  Detail & Related papers  (2023-06-02T12:30:42Z)
• Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Task [49.50140712943701]
  We evaluate the performance of ChatGPT/GPT-4 on a radiology NLI task and compare it to other models fine-tuned specifically on task-related data samples. We also conduct a comprehensive investigation on ChatGPT/GPT-4's reasoning ability by introducing varying levels of inference difficulty.
  arXiv  Detail & Related papers  (2023-04-18T17:21:48Z)
• Energy Transformer [64.22957136952725]
  Our work combines aspects of three promising paradigms in machine learning, namely, attention mechanism, energy-based models, and associative memory. We propose a novel architecture, called the Energy Transformer (or ET for short), that uses a sequence of attention layers that are purposely designed to minimize a specifically engineered energy function.
  arXiv  Detail & Related papers  (2023-02-14T18:51:22Z)
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration [130.89746032163106]
  We propose ALOE, a new algorithm for learning conditional and unconditional EBMs for discrete structured data. We show that the energy function and sampler can be trained efficiently via a new variational form of power iteration. We present an energy model guided fuzzer for software testing that achieves comparable performance to well engineered fuzzing engines like libfuzzer.
  arXiv  Detail & Related papers  (2020-11-10T19:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.