Carbon-Taxed Transformers: A Green Compression Pipeline for Overgrown Language Models

• URL: http://arxiv.org/abs/2604.25903v1
• Date: Tue, 28 Apr 2026 17:48:16 GMT
• Title: Carbon-Taxed Transformers: A Green Compression Pipeline for Overgrown Language Models
• Abstract summary: Carbon-Taxed Transformers (CTT) is a systematic multi-architectural compression principled pipeline ordering.<n>CTT operationalizes a computational carbon tax that penalizes architectural inefficiencies and rewards deployment-ready compression.<n>Our results show that CTT delivers on inference: (1) up to 49x memory reduction, (2) time reduction up to 8-10x for clone detection, up to 3x for summarization, and 4-7x for generation.
• Score: 9.374717320061988
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The accelerating adoption of Large Language Models (LLMs) in software engineering (SE) has brought with it a silent crisis: unsustainable computational cost. While these models demonstrate remarkable capabilities in different SE tasks, they are unmanageably large, slow to deploy, memory-intensive, and carbon-heavy. This reality threatens not only the scalability and accessibility of AI-powered SE, but also its long-term environmental sustainability. The research challenge is clear: we must go beyond accuracy and address efficiency and environmental cost as first-class design constraints. To meet this challenge, we introduce Carbon-Taxed Transformers (CTT), a systematic multi-architectural compression principled pipeline ordering inspired by economic carbon taxation principles. Drawing from the economic concept of carbon pricing, CTT operationalizes a computational carbon tax that penalizes architectural inefficiencies and rewards deployment-ready compression. We evaluate CTT across three core SE tasks: code clone detection, code summarization, and code generation, with models spanning encoder-only, encoder-decoder, and decoder-only architecture. Our results show that CTT delivers on inference: (1) up to 49x memory reduction, (2) time reduction up to 8-10x for clone detection, up to 3x for summarization, and 4-7x for generation, (3) up to 81% reduction in CO2 emissions and (4) CTT retains around 98% accuracy on clone detection, around 89% on summarization, and up to 91% (textual metrics) and 68% (pass@1) for generation. Two ablation studies show that pipeline ordering and individual component contributions are both essential, providing empirical justification for CTT's design and effectiveness. This work establishes a viable path toward responsible AI in SE through aggressive yet performance-preserving compression.

Related papers

• Eco-SoC: A Sustainable VLSI Architecture for Energy-Proportional Artificial Intelligence [0.0]
  This paper introduces Eco-SoC, a highly scalable VLSI architecture co-designed specifically for sustainable artificial intelligence.<n>We propose a hardware-level Dynamic Precision-Scaling Logic (DPSL) framework that adaptively modulates bit-width precision.<n>By introducing a thermal-aware power gating mechanism, Eco-SoC doubles the projected Mean Time To Failure (MTTF) of the silicon.
  arXiv  Detail & Related papers  (2026-08-09T15:21:07Z)
• Evaluating the Environmental Impact of using SLMs and Prompt Engineering for Code Generation [2.2165845825140695]
  This paper presents the first systematic empirical study investigating how different prompt engineering strategies impact code generation accuracy alongside sustainability factors.<n>We evaluate six prominent prompting strategies across 11 open-source models (ranging from 1B to 34B parameters) using the HumanEval+ and MBPP+ benchmarks.<n>Our findings indicate that Chain-of-Thought, being a simpler prompting technique, can provide a near-optimal balance between reasoning capability and energy efficiency.
  arXiv  Detail & Related papers  (2026-04-03T06:37:51Z)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression [55.63153956934198]
  Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs)<n>Existing CoT compression methods often suffer from a critical loss of logical fidelity at high compression ratios.<n>We propose a novel EXTreme-RAtio Chain-of-Thought Compression framework, termed Extra-CoT, which aggressively reduces the token budget while preserving answer accuracy.
  arXiv  Detail & Related papers  (2026-02-09T06:57:15Z)
• Towards Green AI: Decoding the Energy of LLM Inference in Software Development [46.879983975894135]
  AI-assisted tools are increasingly integrated into software development, but their reliance on large language models (LLMs) introduces substantial computational and energy costs.<n>We conduct a phase-level analysis of LLM inference energy consumption, distinguishing between the (1) prefill, where the model processes the input and builds internal representations, and (2) decoding, where output tokens are generated using the stored state.
  arXiv  Detail & Related papers  (2026-02-05T14:38:19Z)
• EcoSpa: Efficient Transformer Training with Coupled Sparsity [79.5008521101473]
  Transformers have become the backbone of modern AI, yet their high computational demands pose critical system challenges.<n>We introduce EcoSpa, an efficient structured sparse training method that jointly evaluates and sparsifies coupled weight matrix pairs.
  arXiv  Detail & Related papers  (2025-11-09T11:23:43Z)
• Carbon-Aware Orchestration of Integrated Satellite Aerial Terrestrial Networks via Digital Twin [6.097337826307388]
  Integrated Satellite Aerial Terrestrial Networks (ISATNs) are envisioned as key enablers of 6G.<n>This work advances prior energy-aware studies by proposing a carbon-aware orchestration framework for ISATNs.
  arXiv  Detail & Related papers  (2025-10-01T06:49:42Z)
• CarboFormer: A Lightweight Semantic Segmentation Architecture for Efficient Carbon Dioxide Detection Using Optical Gas Imaging [4.567122178196833]
  Carbon dioxide (CO$$) emissions are critical indicators of both environmental impact and industrial processes.<n>We introduce CarboFormer, a lightweight semantic segmentation framework for Optical Gas Imaging (OGI)<n>Our approach integrates an optimized encoder-decoder architecture with specialized multi-scale feature fusion and auxiliary supervision strategies.
  arXiv  Detail & Related papers  (2025-05-23T18:01:42Z)
• Accelerating Error Correction Code Transformers [56.75773430667148]
  We introduce a novel acceleration method for transformer-based decoders. We achieve a 90% compression ratio and reduce arithmetic operation energy consumption by at least 224 times on modern hardware.
  arXiv  Detail & Related papers  (2024-10-08T11:07:55Z)
• Generative AI for Low-Carbon Artificial Intelligence of Things with Large Language Models [67.0243099823109]
  Generative AI (GAI) holds immense potential to reduce carbon emissions of Artificial Intelligence of Things (AIoT) In this article, we explore the potential of GAI for carbon emissions reduction and propose a novel GAI-enabled solution for low-carbon AIoT. We propose a Large Language Model (LLM)-enabled carbon emission optimization framework, in which we design pluggable LLM and Retrieval Augmented Generation (RAG) modules.
  arXiv  Detail & Related papers  (2024-04-28T05:46:28Z)
• LACS: Learning-Augmented Algorithms for Carbon-Aware Resource Scaling with Uncertain Demand [1.423958951481749]
  This paper studies the online carbon-aware resource scaling problem with unknown job lengths (OCSU) We propose LACS, a theoretically robust learning-augmented algorithm that solves OCSU. LACS achieves a 32% reduction in carbon footprint compared to the deadline-aware carbon-agnostic execution of the job.
  arXiv  Detail & Related papers  (2024-03-29T04:54:22Z)
• Non-Autoregressive Transformer ASR with CTC-Enhanced Decoder Input [54.82369261350497]
  We propose a CTC-enhanced NAR transformer, which generates target sequence by refining predictions of the CTC module. Experimental results show that our method outperforms all previous NAR counterparts and achieves 50x faster decoding speed than a strong AR baseline with only 0.0 0.3 absolute CER degradation on Aishell-1 and Aishell-2 datasets.
  arXiv  Detail & Related papers  (2020-10-28T15:00:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.