CASHformer: Cognition Aware SHape Transformer for Longitudinal Analysis
- URL: http://arxiv.org/abs/2207.02091v1
- Date: Tue, 5 Jul 2022 14:50:21 GMT
- Title: CASHformer: Cognition Aware SHape Transformer for Longitudinal Analysis
- Authors: Ignacio Sarasua, Sebastian P\"olsterl, Christian Wachinger
- Abstract summary: CASHformer is a transformer-based framework to model longitudinal shape trajectories in Alzheimer's disease.
It reduces the number of parameters by over 90% with respect to the original model.
Our results show that CASHformer reduces the reconstruction error by 73% compared to previously proposed methods.
- Score: 3.7814216736076434
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Modeling temporal changes in subcortical structures is crucial for a better
understanding of the progression of Alzheimer's disease (AD). Given their
flexibility to adapt to heterogeneous sequence lengths, mesh-based transformer
architectures have been proposed in the past for predicting hippocampus
deformations across time. However, one of the main limitations of transformers
is the large amount of trainable parameters, which makes the application on
small datasets very challenging. In addition, current methods do not include
relevant non-image information that can help to identify AD-related patterns in
the progression. To this end, we introduce CASHformer, a transformer-based
framework to model longitudinal shape trajectories in AD. CASHformer
incorporates the idea of pre-trained transformers as universal compute engines
that generalize across a wide range of tasks by freezing most layers during
fine-tuning. This reduces the number of parameters by over 90% with respect to
the original model and therefore enables the application of large models on
small datasets without overfitting. In addition, CASHformer models cognitive
decline to reveal AD atrophy patterns in the temporal sequence. Our results
show that CASHformer reduces the reconstruction error by 73% compared to
previously proposed methods. Moreover, the accuracy of detecting patients
progressing to AD increases by 3% with imputing missing longitudinal shape
data.
Related papers
- Few-Shot Class Incremental Learning via Robust Transformer Approach [16.590193619691416]
Few-Shot Class-Incremental Learning presents an extension of the Class Incremental Learning problem where a model is faced with the problem of data scarcity.
This problem remains an open problem because all recent works are built upon the convolutional neural networks performing sub-optimally.
Our paper presents Robust Transformer Approach built upon the Compact Convolution Transformer.
arXiv Detail & Related papers (2024-05-08T03:35:52Z) - HAFFormer: A Hierarchical Attention-Free Framework for Alzheimer's Disease Detection From Spontaneous Speech [42.688549469089985]
We construct a novel framework, namely Hierarchical Attention-Free Transformer (HAFFormer), to better deal with long speech for Alzheimer's Disease detection.
Specifically, we employ an attention-free module of Multi-Scale Depthwise Convolution to replace the self-attention and thus avoid the expensive computation.
By conducting extensive experiments on the ADReSS-M dataset, the introduced HAFFormer can achieve competitive results (82.6% accuracy) with other recent work.
arXiv Detail & Related papers (2024-05-07T02:19:16Z) - DiffusionPCR: Diffusion Models for Robust Multi-Step Point Cloud
Registration [73.37538551605712]
Point Cloud Registration (PCR) estimates the relative rigid transformation between two point clouds.
We propose formulating PCR as a denoising diffusion probabilistic process, mapping noisy transformations to the ground truth.
Our experiments showcase the effectiveness of our DiffusionPCR, yielding state-of-the-art registration recall rates (95.3%/81.6%) on 3D and 3DLoMatch.
arXiv Detail & Related papers (2023-12-05T18:59:41Z) - The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease
detection [51.697248252191265]
This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation.
We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare.
Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
arXiv Detail & Related papers (2023-09-13T10:40:41Z) - Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of
Language Model [92.55145016562867]
We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance.
Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
arXiv Detail & Related papers (2023-05-24T15:52:08Z) - Posterior Collapse and Latent Variable Non-identifiability [54.842098835445]
We propose a class of latent-identifiable variational autoencoders, deep generative models which enforce identifiability without sacrificing flexibility.
Across synthetic and real datasets, latent-identifiable variational autoencoders outperform existing methods in mitigating posterior collapse and providing meaningful representations of the data.
arXiv Detail & Related papers (2023-01-02T06:16:56Z) - Accurate and Robust Lesion RECIST Diameter Prediction and Segmentation
with Transformers [22.528235432455524]
This paper proposes a transformer-based network for lesion RECIST diameter prediction and segmentation (LRDPS)
It is formulated as three correlative and complementary tasks: lesion segmentation, heatmap prediction, and keypoint regression.
MeaFormer achieves the state-of-the-art performance of LRDPS on the large-scale DeepLesion dataset.
arXiv Detail & Related papers (2022-08-28T01:43:21Z) - ER: Equivariance Regularizer for Knowledge Graph Completion [107.51609402963072]
We propose a new regularizer, namely, Equivariance Regularizer (ER)
ER can enhance the generalization ability of the model by employing the semantic equivariance between the head and tail entities.
The experimental results indicate a clear and substantial improvement over the state-of-the-art relation prediction methods.
arXiv Detail & Related papers (2022-06-24T08:18:05Z) - Entropy optimized semi-supervised decomposed vector-quantized
variational autoencoder model based on transfer learning for multiclass text
classification and generation [3.9318191265352196]
We propose a semisupervised discrete latent variable model for multi-class text classification and text generation.
The proposed model employs the concept of transfer learning for training a quantized transformer model.
Experimental results indicate that the proposed model has surpassed the state-of-the-art models remarkably.
arXiv Detail & Related papers (2021-11-10T07:07:54Z) - Disentangling Generative Factors of Physical Fields Using Variational
Autoencoders [0.0]
This work explores the use of variational autoencoders (VAEs) for non-linear dimension reduction.
A disentangled decomposition is interpretable and can be transferred to a variety of tasks including generative modeling.
arXiv Detail & Related papers (2021-09-15T16:02:43Z) - Discrete Auto-regressive Variational Attention Models for Text Modeling [53.38382932162732]
Variational autoencoders (VAEs) have been widely applied for text modeling.
They are troubled by two challenges: information underrepresentation and posterior collapse.
We propose Discrete Auto-regressive Variational Attention Model (DAVAM) to address the challenges.
arXiv Detail & Related papers (2021-06-16T06:36:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.