MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance

• URL: http://arxiv.org/abs/2412.10730v1
• Date: Sat, 14 Dec 2024 07:58:24 GMT
• Title: MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance
• Authors: Wenjun Huang, Jianguo Hu,
• Abstract summary: We introduce MAL (Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance), a novel framework that enhances xLSTM's capabilities through innovative pretraining strategies. We propose a cluster-masked masking method that significantly improves local feature capture and optimize image scanning efficiency. Our universal encoder-decoder pretraining approach integrates multiple tasks, including image autoregression, depth estimation, and image segmentation, thereby enhancing the model's adaptability and robustness across diverse visual tasks.
• Score: 2.45239928345171
• License:
• Abstract: The Long Short-Term Memory (LSTM) networks have traditionally faced challenges in scaling and effectively capturing complex dependencies in visual tasks. The xLSTM architecture has emerged to address these limitations, incorporating exponential gating and a parallel matrix memory structure to enhance performance and scalability. Despite these advancements, the potential of xLSTM in visual computing has not been fully realized, particularly in leveraging autoregressive techniques for improved feature extraction. In this paper, we introduce MAL (Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance), a novel framework that enhances xLSTM's capabilities through innovative pretraining strategies. We propose a cluster-masked masking method that significantly improves local feature capture and optimizes image scanning efficiency. Additionally, our universal encoder-decoder pretraining approach integrates multiple tasks, including image autoregression, depth estimation, and image segmentation, thereby enhancing the model's adaptability and robustness across diverse visual tasks. Our experimental results demonstrate that MAL surpasses traditional supervised models and fully leverages the scaling potential of xLSTM, setting a new benchmark in visual task performance.

Related papers

• Improving Autoregressive Visual Generation with Cluster-Oriented Token Prediction [52.09472099976885]
  IAR is an Improved AutoRegressive Visual Generation Method. We propose a Codebook Rearrangement strategy that uses balanced k-means clustering algorithm. We also propose a Cluster-oriented Cross-entropy Loss that guides the model to correctly predict the cluster where the token is located.
  arXiv  Detail & Related papers  (2025-01-01T15:58:51Z)
• Selective State Space Memory for Large Vision-Language Models [0.0]
  State Space Memory Integration (SSMI) is a novel approach for efficient fine-tuning of LVLMs. SSMI captures long-range dependencies and injects task-specific visual and sequential patterns effectively. experiments on benchmark datasets, including COCO Captioning, VQA, and Flickr30k, demonstrate that SSMI achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-13T05:40:50Z)
• SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding [66.74446220401296]
  We propose SynerGen-VL, a simple yet powerful encoder-free MLLM capable of both image understanding and generation. We introduce the token folding mechanism and the vision-expert-based progressive alignment pretraining strategy, which effectively support high-resolution image understanding. Our code and models shall be released.
  arXiv  Detail & Related papers  (2024-12-12T18:59:26Z)
• MAT: Multi-Range Attention Transformer for Efficient Image Super-Resolution [14.265237560766268]
  A flexible integration of attention across diverse spatial extents can yield significant performance enhancements. We introduce Multi-Range Attention Transformer (MAT) tailored for Super Resolution (SR) tasks. MAT adeptly capture dependencies across various spatial ranges, improving the diversity and efficacy of its feature representations.
  arXiv  Detail & Related papers  (2024-11-26T08:30:31Z)
• xLSTM-FER: Enhancing Student Expression Recognition with Extended Vision Long Short-Term Memory Network [0.8287206589886881]
  This paper introduces xLSTM-FER, a novel architecture derived from the Extended Long Short-Term Memory (xLSTM) xLSTM-FER processes input images by segmenting them into a series of patches and leveraging a stack of xLSTM blocks to handle these patches. Experiments on CK+, RAF-DF, and FERplus demonstrate the potential of xLSTM-FER in expression recognition tasks.
  arXiv  Detail & Related papers  (2024-10-07T14:29:24Z)
• HRVMamba: High-Resolution Visual State Space Model for Dense Prediction [60.80423207808076]
  State Space Models (SSMs) with efficient hardware-aware designs have demonstrated significant potential in computer vision tasks. These models have been constrained by three key challenges: insufficient inductive bias, long-range forgetting, and low-resolution output representation. We introduce the Dynamic Visual State Space (DVSS) block, which employs deformable convolution to mitigate the long-range forgetting problem. We also introduce High-Resolution Visual State Space Model (HRVMamba) based on the DVSS block, which preserves high-resolution representations throughout the entire process.
  arXiv  Detail & Related papers  (2024-10-04T06:19:29Z)
• MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model [49.931663904599205]
  MaVEn is an innovative framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. We show that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.
  arXiv  Detail & Related papers  (2024-08-22T11:57:16Z)
• Cross-Scan Mamba with Masked Training for Robust Spectral Imaging [51.557804095896174]
  We propose the Cross-Scanning Mamba, named CS-Mamba, that employs a Spatial-Spectral SSM for global-local balanced context encoding. Experiment results show that our CS-Mamba achieves state-of-the-art performance and the masked training method can better reconstruct smooth features to improve the visual quality.
  arXiv  Detail & Related papers  (2024-08-01T15:14:10Z)
• Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model [26.786890883280062]
  State Space Models (SSMs) have garnered widespread attention due to their global receptive field and linear complexity. To improve the performance of SSMs in vision tasks, a multi-scan strategy is widely adopted. We introduce Multi-Scale Vision Mamba (MSVMamba) to preserve the superiority of SSMs in vision tasks with limited parameters.
  arXiv  Detail & Related papers  (2024-05-23T04:59:49Z)
• Object Tracking through Residual and Dense LSTMs [67.98948222599849]
  Deep learning-based trackers based on LSTMs (Long Short-Term Memory) recurrent neural networks have emerged as a powerful alternative. DenseLSTMs outperform Residual and regular LSTM, and offer a higher resilience to nuisances. Our case study supports the adoption of residual-based RNNs for enhancing the robustness of other trackers.
  arXiv  Detail & Related papers  (2020-06-22T08:20:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.