Hybrid Reinforced Medical Report Generation with M-Linear Attention and Repetition Penalty

• URL: http://arxiv.org/abs/2210.13729v1
• Date: Fri, 14 Oct 2022 15:27:34 GMT
• Title: Hybrid Reinforced Medical Report Generation with M-Linear Attention and Repetition Penalty
• Authors: Wenting Xu, Zhenghua Xu, Junyang Chen, Chang Qi, Thomas Lukasiewicz
• Abstract summary: We propose a hybrid reinforced medical report generation method with m-linear attention and repetition penalty mechanism. Specifically, a hybrid reward with different weights is employed to remedy the limitations of single-metric-based rewards. We also propose a search algorithm with linear complexity to approximate the best weight combination.
• Score: 45.92216112110279
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To reduce doctors' workload, deep-learning-based automatic medical report generation has recently attracted more and more research efforts, where deep convolutional neural networks (CNNs) are employed to encode the input images, and recurrent neural networks (RNNs) are used to decode the visual features into medical reports automatically. However, these state-of-the-art methods mainly suffer from three shortcomings: (i) incomprehensive optimization, (ii) low-order and unidimensional attention mechanisms, and (iii) repeated generation. In this article, we propose a hybrid reinforced medical report generation method with m-linear attention and repetition penalty mechanism (HReMRG-MR) to overcome these problems. Specifically, a hybrid reward with different weights is employed to remedy the limitations of single-metric-based rewards. We also propose a search algorithm with linear complexity to approximate the best weight combination. Furthermore, we use m-linear attention modules to explore high-order feature interactions and to achieve multi-modal reasoning, while a repetition penalty applies penalties to repeated terms during the model's training process. Extensive experimental studies on two public datasets show that HReMRG-MR greatly outperforms the state-of-the-art baselines in terms of all metrics. We also conducted a series of ablation experiments to prove the effectiveness of all our proposed components. We also performed a reward search toy experiment to give evidence that our proposed search approach can significantly reduce the search time while approximating the best performance.

Related papers

• Fine-tuning -- a Transfer Learning approach [0.22344294014777952]
  Missingness in Electronic Health Records (EHRs) is often hampered by the abundance of missing data in this valuable resource. Existing deep imputation methods rely on end-to-end pipelines that incorporate both imputation and downstream analyses. This paper explores the development of a modular, deep learning-based imputation and classification pipeline.
  arXiv  Detail & Related papers  (2024-11-06T14:18:23Z)
• Copy Recurrent Neural Network Structure Network [0.0]
  We propose a novel ICD path generation framework called Copy Recurrent Neural Network Structure Network (CRNNet) By using RNNs to generate sequential outputs and incorporating a copy module, we efficiently identify complication diseases. Our method achieves a 57.30% ratio of complex diseases in predictions, outperforming state-of-the-art and previous approaches.
  arXiv  Detail & Related papers  (2023-05-22T17:22:37Z)
• A Direct Sampling-Based Deep Learning Approach for Inverse Medium Scattering Problems [3.776050336003086]
  We propose a novel direct sampling-based deep learning approach (DSM-DL) for reconstructing inhomogeneous scatterers. Our proposed DSM-DL is computationally efficient, robust to noise, easy to implement, and able to naturally incorporate multiple measured data.
  arXiv  Detail & Related papers  (2023-04-29T12:29:30Z)
• Deep AUC Maximization for Medical Image Classification: Challenges and Opportunities [60.079782224958414]
  We will present and discuss opportunities and challenges brought by a new deep learning method by AUC (aka underlinebf Deep underlinebf AUC classification)
  arXiv  Detail & Related papers  (2021-11-01T15:31:32Z)
• Complex-valued Federated Learning with Differential Privacy and MRI Applications [51.34714485616763]
  We introduce the complex-valued Gaussian mechanism, whose behaviour we characterise in terms of $f$-DP, $(varepsilon, delta)$-DP and R'enyi-DP. We present novel complex-valued neural network primitives compatible with DP. Experimentally, we showcase a proof-of-concept by training federated complex-valued neural networks with DP on a real-world task.
  arXiv  Detail & Related papers  (2021-10-07T14:03:00Z)
• DFENet: A Novel Dimension Fusion Edge Guided Network for Brain MRI Segmentation [0.0]
  We propose a novel Dimension Fusion Edge-guided network (DFENet) that can meet both of these requirements by fusing the features of 2D and 3D CNNs. The proposed model is robust, accurate, superior to the existing methods, and can be relied upon for biomedical applications.
  arXiv  Detail & Related papers  (2021-05-17T15:43:59Z)
• Reinforced Medical Report Generation with X-Linear Attention and Repetition Penalty [46.51332238677608]
  We propose a reinforced medical report generation solution with x-linear attention and repetition penalty mechanisms. x-linear attention modules are used to explore high-order feature interactions and achieve multi-modal reasoning. ReMRG-XR greatly outperforms the state-of-the-art baselines in terms of all metrics.
  arXiv  Detail & Related papers  (2020-11-16T01:44:47Z)
• AutoHR: A Strong End-to-end Baseline for Remote Heart Rate Measurement with Neural Searching [76.4844593082362]
  We investigate the reason why existing end-to-end networks perform poorly in challenging conditions and establish a strong baseline for remote HR measurement with architecture search (NAS) Comprehensive experiments are performed on three benchmark datasets on both intra-temporal and cross-dataset testing.
  arXiv  Detail & Related papers  (2020-04-26T05:43:21Z)
• MS-Net: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Data [75.73881040581767]
  We propose a novel multi-site network (MS-Net) for improving prostate segmentation by learning robust representations. Our MS-Net improves the performance across all datasets consistently, and outperforms state-of-the-art methods for multi-site learning.
  arXiv  Detail & Related papers  (2020-02-09T14:11:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.