DSReLU: A Novel Dynamic Slope Function for Superior Model Training

• URL: http://arxiv.org/abs/2408.09156v1
• Date: Sat, 17 Aug 2024 10:01:30 GMT
• Title: DSReLU: A Novel Dynamic Slope Function for Superior Model Training
• Authors: Archisman Chakraborti, Bidyut B Chaudhuri,
• Abstract summary: The rationale behind this approach is to overcome limitations associated with traditional activation functions, such as ReLU. Evaluated on the Mini-ImageNet, CIFAR-100, and MIT-BIH datasets, our method demonstrated improvements in classification metrics and generalization capabilities.
• Score: 2.2057562301812674
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study introduces a novel activation function, characterized by a dynamic slope that adjusts throughout the training process, aimed at enhancing adaptability and performance in deep neural networks for computer vision tasks. The rationale behind this approach is to overcome limitations associated with traditional activation functions, such as ReLU, by providing a more flexible mechanism that can adapt to different stages of the learning process. Evaluated on the Mini-ImageNet, CIFAR-100, and MIT-BIH datasets, our method demonstrated improvements in classification metrics and generalization capabilities. These results suggest that our dynamic slope activation function could offer a new tool for improving the performance of deep learning models in various image recognition tasks.

Related papers

• Efficient Search for Customized Activation Functions with Gradient Descent [42.20716255578699]
  Different activation functions work best for different deep learning models. We propose a fine-grained search cell that combines basic mathematical operations to model activation functions. Our approach enables the identification of specialized activations, leading to improved performance in every model we tried.
  arXiv  Detail & Related papers  (2024-08-13T11:27:31Z)
• AggSS: An Aggregated Self-Supervised Approach for Class-Incremental Learning [17.155759991260094]
  This paper investigates the impact of self-supervised learning, specifically image rotations, on various class-incremental learning paradigms. We observe a shift in the deep neural network's attention towards intrinsic object features as it learns through AggSS strategy. AggSS serves as a plug-and-play module that can be seamlessly incorporated into any class-incremental learning framework.
  arXiv  Detail & Related papers  (2024-08-08T10:16:02Z)
• Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learning [113.89327264634984]
  Few-shot class-incremental learning (FSCIL) confronts the challenge of integrating new classes into a model with minimal training samples. Traditional methods widely adopt static adaptation relying on a fixed parameter space to learn from data that arrive sequentially. We propose a dual selective SSM projector that dynamically adjusts the projection parameters based on the intermediate features for dynamic adaptation.
  arXiv  Detail & Related papers  (2024-07-08T17:09:39Z)
• Learning Objective-Specific Active Learning Strategies with Attentive Neural Processes [72.75421975804132]
  Learning Active Learning (LAL) suggests to learn the active learning strategy itself, allowing it to adapt to the given setting. We propose a novel LAL method for classification that exploits symmetry and independence properties of the active learning problem. Our approach is based on learning from a myopic oracle, which gives our model the ability to adapt to non-standard objectives.
  arXiv  Detail & Related papers  (2023-09-11T14:16:37Z)
• Understanding Self-attention Mechanism via Dynamical System Perspective [58.024376086269015]
  Self-attention mechanism (SAM) is widely used in various fields of artificial intelligence. We show that intrinsic stiffness phenomenon (SP) in the high-precision solution of ordinary differential equations (ODEs) also widely exists in high-performance neural networks (NN) We show that the SAM is also a stiffness-aware step size adaptor that can enhance the model's representational ability to measure intrinsic SP.
  arXiv  Detail & Related papers  (2023-08-19T08:17:41Z)
• ENN: A Neural Network with DCT Adaptive Activation Functions [2.2713084727838115]
  We present Expressive Neural Network (ENN), a novel model in which the non-linear activation functions are modeled using the Discrete Cosine Transform (DCT) This parametrization keeps the number of trainable parameters low, is appropriate for gradient-based schemes, and adapts to different learning tasks. The performance of ENN outperforms state of the art benchmarks, providing above a 40% gap in accuracy in some scenarios.
  arXiv  Detail & Related papers  (2023-07-02T21:46:30Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• FOSTER: Feature Boosting and Compression for Class-Incremental Learning [52.603520403933985]
  Deep neural networks suffer from catastrophic forgetting when learning new categories. We propose a novel two-stage learning paradigm FOSTER, empowering the model to learn new categories adaptively.
  arXiv  Detail & Related papers  (2022-04-10T11:38:33Z)
• Discovering Parametric Activation Functions [17.369163074697475]
  This paper proposes a technique for customizing activation functions automatically, resulting in reliable improvements in performance. Experiments with four different neural network architectures on the CIFAR-10 and CIFAR-100 image classification datasets show that this approach is effective.
  arXiv  Detail & Related papers  (2020-06-05T00:25:33Z)
• Dynamic Memory Induction Networks for Few-Shot Text Classification [84.88381813651971]
  This paper proposes Dynamic Memory Induction Networks (DMIN) for few-shot text classification. The proposed model achieves new state-of-the-art results on the miniRCV1 and ODIC dataset, improving the best performance (accuracy) by 24%.
  arXiv  Detail & Related papers  (2020-05-12T12:41:14Z)
• Evolutionary Optimization of Deep Learning Activation Functions [15.628118691027328]
  We show that evolutionary algorithms can discover novel activation functions that outperform the Rectified Linear Unit (ReLU) replacing ReLU with evolved activation functions results in statistically significant increases in network accuracy. These novel activation functions are shown to generalize, achieving high performance across tasks.
  arXiv  Detail & Related papers  (2020-02-17T19:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.