Recent advances in deep learning have shown great potential for the automatic generation of medical imaging reports. Deep learning techniques, inspired by image captioning, have made significant progress in the field of diagnostic report generation. This paper provides a comprehensive overview of recent research efforts in deep learning-based medical imaging report generation and proposes future directions in this field. First, we summarize and analyze the data set, architecture, application, and evaluation of deep learning-based medical imaging report generation. Specially, we survey the deep learning architectures used in diagnostic report generation, including hierarchical RNN-based frameworks, attention-based frameworks, and reinforcement learning-based frameworks. In addition, we identify potential challenges and suggest future research directions to support clinical applications and decision-making using medical imaging report generation systems. 

/pdf/a-survey-on-automatic-generation-of-medical-imaging-reports-1664cbgo.pdf

A survey on automatic generation of medical imaging reports based on deep learning

Medical image captioning plays an important role in modern healthcare, improving clinical report generation and aiding radiologists in detecting abnormalities and reducing misdiagnosis. The complex visual and textual data biases make this task more challenging. Recent advancements in transformer-based models have significantly improved the generation of radiology reports from medical images. However, these models require substantial computational resources for training and have been observed to produce unnatural language outputs when trained solely on raw image-text pairs. Our aim is to generate more detailed reports specific to images and to explain the reasoning behind the generated text through image-text alignment. Given the high computational demands of end-to-end model training, we introduce a two-step training methodology with an Intelligent Visual Encoder for Bridging Modalities in Report Generation (InVERGe) model. This model incorporates a lightweight transformer known as the Cross-Modal Query Fusion Layer (CMQFL), which utilizes the output from a frozen encoder to identify the most relevant text-grounded image embedding. This layer bridges the gap between the encoder and decoder, significantly reducing the workload on the decoder and enhancing the alignment between vision and language. Our experimental results, conducted using the MIMIC-CXR, Indiana University chest X-ray images, and CDD-CESM breast images datasets, demonstrate the effectiveness of our approach. Code: https://github.com/labsroy007/InVERGe

InVERGe: Intelligent Visual Encoder for Bridging Modalities in Report Generation

Reconstructing cone-beam computed tomography (CBCT) typically utilizes a Feldkamp-Davis-Kress (FDK) algorithm to ‘translate’ hundreds of 2D X-ray projections on different angles into a 3D CT image. For minimizing the X-ray induced ionizing radiation, sparse-view CBCT takes fewer projections by a wider-angle interval, but suffers from an inferior CT reconstruction quality. To solve this, the recent solutions mainly resort to synthesizing missing projections, and force the synthesized projections to be as realistic as those actual ones, which is extremely difficult due to X-ray’s tissue superimposing. In this paper, we argue that the synthetic projections should restore FDK-required information as much as possible, while the visual fidelity is the secondary importance. Inspired by a simple fact that FDK only relies on frequency information after ramp-filtering for reconstruction, we develop a Reconstruction-Friendly Interpolation Network (RFI-Net), which first utilizes a 3D-2D attention network to learn inter-projection relations for synthesizing missing projections, and then introduces a novel Ramp-Filter loss to constrain a frequency consistency between the synthesized and real projections after ramp-filtering. By doing so, RFI-Net’s energy can be forcibly devoted to restoring more CT-reconstruction useful information in projection synthesis. We build a complete reconstruction framework consisting of our developed RFI-Net, FDK and a commonly-used CT post-refinement. Experimental results on reconstruction from only one-eighth projections demonstrate that using RFI-Net restored full-view projections can significantly improve the reconstruction quality by increasing PSNR by 2.59 dB and 2.03 dB on the walnut and patient CBCT datasets, respectively, comparing with using those restored by other state-of-the-arts. 

Improving the Quality of Sparse-view Cone-Beam Computed Tomography via Reconstruction-Friendly Interpolation Network

Analyzing medical records takes a large amount of time and resources for medical workers. Therefore, the advancement of report generation technology is critical for saving doctors’ time while producing reports. Some diseases frequently exhibit co-occurrence interactions in the context of medical picture categorization and report creation, which necessitate hu-man-designed and added representation. To further capture this latent information, we propose creating a disease co-occurrence matrix from the existing dataset and training a graph neural network on it. We use multi-label contrastive learning to assist the model in distinguishing the mutual links between different diseases. Meanwhile, we incorporate report retrieval module and decoder model to complete the report generation task, and the multi-task learning of disease classification and report generation can improve the generalization ability of the target task. Experimental results show that our pro-posed combined method and multi-task learning approach have shown significant improvement compared to previous research.

Multi-Task Learning for Lung Disease Classification and Report Generation via Prior Graph Structure and Contrastive Learning

Generating coherent reports from medical images is an important task for reducing doctors' workload. Unlike traditional image captioning tasks, the task of medical image report generation faces more challenges. Current models for generating reports from medical images often fail to characterize some abnormal findings, and some models generate reports with low quality. In this study, we propose a model to generate high-quality reports from medical images.In this paper, we propose a model called Hybrid Discriminator Generative Adversarial Network (HDGAN), which combines Generative Adversarial Network (GAN) with Reinforcement Learning (RL). The HDGAN model consists of a generator, a one-sentence discriminator, and a one-word discriminator. Specifically, the RL reward signals are judged on the one-sentence discriminator and one-word discriminator separately. The one-sentence discriminator can better learn sentence-level structural information, while the one-word discriminator can learn word diversity information effectively.Our approach performs better on the IU-X-ray and COV-CTR datasets than the baseline models. For the ROUGE metric, our method outperforms the state-of-the-art model by 0.36 on the IU-X-ray, 0.06 on the MIMIC-CXR and 0.156 on the COV-CTR.The compositional framework we proposed can generate more accurate medical image reports at different levels. 

Hierarchical medical image report adversarial generation with hybrid discriminator

Excessive ionizing radiation in cone-beam computed tomography (CBCT) causes damage to patients, whereas a low radiation dose degrades the imaging quality. To improve the quality of low-dose CBCT images, deep-learning-based methods are developed and have obtained good performance. However, most previous studies only process the reconstructed CT images, and cannot recover the structures already lost in the reconstruction process. In this paper, a dual-domain attention-guided network framework (Dual-AGNet) is developed to process images in both projection and reconstruction domains. Spatial attention modules are included in the AGNet to effectively and adaptively compensate the intra- and inter-images information in both domains. Moreover, a joint loss function is developed to circumvent the structures loss and over-smoothness in CT images. Our method is evaluated and compared with the state-of-the-art methods on a simulated and a real low-dose CBCT datasets of walnuts. Our Dual-AGNet obtains significantly better performance than the state-of-the-art methods; on the simulated and real datasets, it decreases the root mean square error by approximately 11% and 19%, increases the peak signal-to-noise ratio by approximately 5% and 7%, and increases the structural similarity by approximately 5% and 2%, respectively. In qualitative evaluation, our Dual-AGNet not only suppresses the noise, but also provides realistic CT images with many delicate structures. In addition, the developed Dual-AGNet can be integrated into the existing CBCT system to promote the development of low-dose CBCT imaging. Testing code is available at https://github.com/LianyingChao/Dual-AGNet . • A novel dual-domain deep learning framework for low-dose CT reconstruction. • A 3D spatial attention module for well utilizing the intra- and inter-images information. • A novel joint loss function for circumventing the structures loss and over-smoothness. • Consistently good performance was obtained on both simulated and real datasets. 

Dual-domain attention-guided convolutional neural network for low-dose cone-beam computed tomography reconstruction

To reduce doctors' workload, deep-learning-based automatic medical report generation has recently attracted more and more research efforts, where attention mechanisms and reinforcement learning are integrated with the classic encoder-decoder architecture to enhance the performance of deep models. However, these state-of-the-art solutions mainly suffer from two shortcomings: (i) their attention mechanisms cannot utilize high-order feature interactions, and (ii) due to the use of TF-IDF-based reward functions, these methods are fragile with generating repeated terms. Therefore, in this work, we propose a reinforced medical report generation solution with x-linear attention and repetition penalty mechanisms (ReMRG-XR) to overcome these problems. Specifically, x-linear attention modules are used to explore high-order feature interactions and achieve multi-modal reasoning, while repetition penalty is used to apply penalties to repeated terms during the model's training process. Extensive experimental studies have been conducted on two public datasets, and the results show that ReMRG-XR greatly outperforms the state-of-the-art baselines in terms of all metrics.

Reinforced Medical Report Generation with X-Linear Attention and Repetition Penalty.

In clinical scenarios, multiple medical images with different views are usually generated at the same time, and they have high semantic consistency. However, the existing medical report generation methods cannot exploit the rich multi-view mutual information of medical images. Therefore, in this work, we propose the first multi-view medical report generation model, called MvCo-DoT. Specifically, MvCo-DoT first propose a multi-view contrastive learning (MvCo) strategy to help the deep reinforcement learning based model utilize the consistency of multi-view inputs for better model learning. Then, to close the performance gaps of using multi-view and single-view inputs, a domain transfer network is further proposed to ensure MvCo-DoT achieve almost the same performance as multi-view inputs using only single-view inputs.Extensive experiments on the IU X-Ray public dataset show that MvCo-DoT outperforms the SOTA medical report generation baselines in all metrics.

pdf/mvco-dot-multi-view-contrastive-domain-transfer-network-for-1ledp7h0.pdf

MvCo-DoT: Multi-View Contrastive Domain Transfer Network for Medical Report Generation

PSDP: Pseudo-supervised dual-processing for low-dose cone-beam computed tomography reconstruction

Surgical action detection based on path aggregation adaptive spatial network

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Wenting Xu

Author Tools

Chat about Author