SHARCS: Efficient Transformers through Routing with Dynamic Width Sub-networks

• URL: http://arxiv.org/abs/2310.12126v1
• Date: Wed, 18 Oct 2023 17:35:15 GMT
• Title: SHARCS: Efficient Transformers through Routing with Dynamic Width Sub-networks
• Abstract summary: We introduce SHARCS for adaptive inference that takes into account the hardness of input samples. SHARCS can train a router on any transformer network, enabling the model to direct different samples to sub-networks with varying widths.
• Score: 93.0460007020805
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce SHARCS for adaptive inference that takes into account the hardness of input samples. SHARCS can train a router on any transformer network, enabling the model to direct different samples to sub-networks with varying widths. Our experiments demonstrate that: (1) SHARCS outperforms or complements existing per-sample adaptive inference methods across various classification tasks in terms of accuracy vs. FLOPs; (2) SHARCS generalizes across different architectures and can be even applied to compressed and efficient transformer encoders to further improve their efficiency; (3) SHARCS can provide a 2 times inference speed up at an insignificant drop in accuracy.

Related papers

• What and Where to Adapt: Structure-Semantics Co-Tuning for Machine Vision Compression via Synergistic Adapters [3.5781276092843393]
  We introduce Structure-Semantics Co-Tuning (S2-CoT), a novel framework that achieves this coordination via two specialized adapters.<n>S2-CoT achieves state-of-the-art results across four diverse base codecs with only a small fraction of trainable parameters.
  arXiv  Detail & Related papers  (2026-04-11T04:03:37Z)
• Efficient Adaptive Transformer: An Empirical Study and Reproducible Framework [0.0]
  EAT provides an open-source benchmarking pipeline that automates data processing, timing, and ablation across GLUE tasks.<n>The main contribution is the open, end-to-end reproducible framework, complete with scripts, CSV logging, and analysis utilities.
  arXiv  Detail & Related papers  (2025-10-14T11:40:48Z)
• Transformer Modeling for Both Scalability and Performance in Multivariate Time Series [0.0]
  We propose a transformer with Delegate Token Attention (DELTAformer) to constrain inter-variable modeling.<n>Our results show that DELTAformer scales linearly with variable-count while actually outperforming standard transformers.
  arXiv  Detail & Related papers  (2025-09-23T18:28:24Z)
• Chain-of-Thought Enhanced Shallow Transformers for Wireless Symbol Detection [14.363929799618283]
  We propose CHain Of thOught Symbol dEtection (CHOOSE), a CoT-enhanced shallow Transformer framework for wireless symbol detection.<n>By introducing autoregressive latent reasoning steps within the hidden space, CHOOSE significantly improves the reasoning capacity of shallow models.<n> Experimental results demonstrate that our approach outperforms conventional shallow Transformers and achieves performance comparable to that of deep Transformers.
  arXiv  Detail & Related papers  (2025-06-26T08:41:45Z)
• UniMoD: Efficient Unified Multimodal Transformers with Mixture-of-Depths [17.68867710994329]
  UniMoD is a task-aware token pruning method that employs a separate router for each task to determine which tokens should be pruned. We apply our method to Show-o and Emu3, reducing training FLOPs by approximately 15% in Show-o and 40% in Emu3, while maintaining or improving performance on several benchmarks.
  arXiv  Detail & Related papers  (2025-02-10T13:52:52Z)
• Adaptive Pruning of Pretrained Transformer via Differential Inclusions [48.47890215458465]
  Current compression algorithms prune transformers at fixed compression ratios, requiring a unique pruning process for each ratio. We propose pruning of pretrained transformers at any desired ratio within a single pruning stage, based on a differential inclusion for a mask parameter. This dynamic can generate the whole regularization solution path of the mask parameter, whose support set identifies the network structure.
  arXiv  Detail & Related papers  (2025-01-06T06:34:52Z)
• iConFormer: Dynamic Parameter-Efficient Tuning with Input-Conditioned Adaptation [15.97351561456467]
  In this paper, we propose a novel PEFT approach, input-Conditioned transFormer, termed iConFormer. We introduce an input-Conditioned Network (iCoN) in the dynamic adapter that enables instance-level feature transformation. To be specific, iCoN generates channel-wise convolutional kernels for each feature and transform it using adaptive convolution process to effectively capture task-specific and fine-grained details tailor to downstream tasks.
  arXiv  Detail & Related papers  (2024-09-04T16:06:23Z)
• Adaptive Step-size Perception Unfolding Network with Non-local Hybrid Attention for Hyperspectral Image Reconstruction [0.39134031118910273]
  We propose an adaptive step-size perception unfolding network (ASPUN), a deep unfolding network based on FISTA algorithm. In addition, we design a Non-local Hybrid Attention Transformer(NHAT) module for fully leveraging the receptive field advantage of transformer. Experimental results show that our ASPUN is superior to the existing SOTA algorithms and achieves the best performance.
  arXiv  Detail & Related papers  (2024-07-04T16:09:52Z)
• Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot [50.16171384920963]
  transformer architecture has prevailed in various deep learning settings. One-layer transformer trained with gradient descent provably learns the sparse token selection task.
  arXiv  Detail & Related papers  (2024-06-11T02:15:53Z)
• Steerable Transformers for Volumetric Data [5.564976582065106]
  We propose an equivariant attention mechanism that operates on features extracted by steerable convolutions.<n>Our experiments in both two and three dimensions show that adding steerable transformer layers to steerable convolutional networks enhances performance.
  arXiv  Detail & Related papers  (2024-05-24T20:43:19Z)
• Sample-based Dynamic Hierarchical Transformer with Layer and Head Flexibility via Contextual Bandit [24.78757412559944]
  Transformer requires a fixed number of layers and heads which makes them inflexible to the complexity of individual samples. We propose a sample-based Dynamic Hierarchical Transformer model whose layers and heads can be dynamically configured with single data samples. We achieve up to 74% computational savings for both training and inference with a minimal loss of accuracy.
  arXiv  Detail & Related papers  (2023-12-05T15:04:11Z)
• Deformable Mixer Transformer with Gating for Multi-Task Learning of Dense Prediction [126.34551436845133]
  CNNs and Transformers have their own advantages and both have been widely used for dense prediction in multi-task learning (MTL) We present a novel MTL model by combining both merits of deformable CNN and query-based Transformer with shared gating for multi-task learning of dense prediction.
  arXiv  Detail & Related papers  (2023-08-10T17:37:49Z)
• 2-D SSM: A General Spatial Layer for Visual Transformers [79.4957965474334]
  A central objective in computer vision is to design models with appropriate 2-D inductive bias. We leverage an expressive variation of the multidimensional State Space Model. Our approach introduces efficient parameterization, accelerated computation, and a suitable normalization scheme.
  arXiv  Detail & Related papers  (2023-06-11T09:41:37Z)
• Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection [88.23337313766353]
  This work first provides a comprehensive statistical theory for transformers to perform ICL. We show that transformers can implement a broad class of standard machine learning algorithms in context. A emphsingle transformer can adaptively select different base ICL algorithms.
  arXiv  Detail & Related papers  (2023-06-07T17:59:31Z)
• Transformers for End-to-End InfoSec Tasks: A Feasibility Study [6.847381178288385]
  We implement transformer models for two distinct InfoSec data formats - specifically URLs and PE files. We show that our URL transformer model requires a different training approach to reach high performance levels. We demonstrate that this approach performs comparably to well-established malware detection models on benchmark PE file datasets.
  arXiv  Detail & Related papers  (2022-12-05T23:50:46Z)
• Rich CNN-Transformer Feature Aggregation Networks for Super-Resolution [50.10987776141901]
  Recent vision transformers along with self-attention have achieved promising results on various computer vision tasks. We introduce an effective hybrid architecture for super-resolution (SR) tasks, which leverages local features from CNNs and long-range dependencies captured by transformers. Our proposed method achieves state-of-the-art SR results on numerous benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-15T06:52:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.