MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
Zizhao Zhang,Yuanpu Xie,Fuyong Xing,Mason McGough,Lin Yang +4 more
- 08 Jul 2017
- pp 3549-3557
TL;DR: This paper proposes MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process.
read more
Abstract: The inability to interpret the model prediction in semantically and visually meaningful ways is a well-known shortcoming of most existing computer-aided diagnosis methods. In this paper, we propose MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process. MDNet includes an image model and a language model. The image model is proposed to enhance multi-scale feature ensembles and utilization efficiency. The language model, integrated with our improved attention mechanism, aims to read and explore discriminative image feature descriptions from reports to learn a direct mapping from sentence words to image pixels. The overall network is trained end-to-end by using our developed optimization strategy. Based on a pathology bladder cancer images and its diagnostic reports (BCIDR) dataset, we conduct sufficient experiments to demonstrate that MDNet outperforms comparative baselines. The proposed image model obtains state-of-the-art performance on two CIFAR datasets as well.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Automatic Report Generation Method for Ultrasound Assisted Diagnosis of Cervical Lymph Nodes
Xuehai Ding,Weili Ren,Yanting Liu,Junjuan Zhao,Chengfan Li,Quan‐Yong Luo,Ce Shen +6 more
- 01 Jan 2023
TL;DR: A novel ultrasound report generation method for the auxiliary diagnosis of cervical lymph nodes is proposed. The method includes a classification network for metastatic prediction and a descriptive sentence generation network for ultrasound characteristics of cervical lymph nodes. The method achieved high accuracy in metastatic finding and description text generation.
•Posted Content
Multiple Instance Captioning: Learning Representations from Histopathology Textbooks and Articles
Jevgenij Gamper,Nasir M. Rajpoot +1 more
TL;DR: ArCH as mentioned in this paper is a computational pathology (CP) multiple instance captioning dataset to facilitate dense supervision of CP tasks, which contains dense diagnostic and morphological descriptions for a range of stains, tissue types and pathologies.
•Posted Content
Learning Efficient, Explainable and Discriminative Representations for Pulmonary Nodules Classification
TL;DR: Wang et al. as mentioned in this paper used neural architecture search (NAS) to automatically search 3D network architectures with excellent accuracy/speed trade-off and used the convolutional block attention module (CBAM) in the networks, which helps us understand the reasoning process.
Factored Attention and Embedding for Unstructured-view Topic-related Ultrasound Report Generation
TL;DR: Both quantitative comparisons and qualitative analysis demonstrate the effectiveness and the superiority of FAE-Gen over seven commonly-used metrics.
Intensive vision-guided network for radiology report generation.
Fudan Zheng,Mengfei Li,Ying Wang,Weijiang Yu,Ruixuan Wang,Zhiguang Chen,Nong Xiao,Yutong Lu +7 more
TL;DR: This work proposes a Globally-intensive Attention (GIA) module in the medical image encoder to simulate and integrate multi-view vision perception, and designs a Visual Knowledge-guided Decoder (VKGD), which can adaptively consider how much the model needs to rely on visual information and previously predicted text to assist next word prediction.
References
Deep Residual Learning for Image Recognition
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
•Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
- 04 Sep 2014
TL;DR: This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
102.6K
Long short-term memory
TL;DR: A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
99K
Going deeper with convolutions
Christian Szegedy,Wei Liu,Yangqing Jia,Pierre Sermanet,Scott Reed,Dragomir Anguelov,Dumitru Erhan,Vincent Vanhoucke,Andrew Rabinovich +8 more
- 07 Jun 2015
TL;DR: Inception as mentioned in this paper is a deep convolutional neural network architecture that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14).
•Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
- 01 Jan 2015
TL;DR: In this paper, the authors investigated the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting and showed that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 layers.
51.9K
Related Papers (5)
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016
Karen Simonyan,Andrew Zisserman +1 more
- 04 Sep 2014