Fetal cardiac ultrasound is a valuable tool for screening fetal cardiac health during pregnancy. Ultrasound standard section testing is an essential part of fetal heart ultrasound diagnosis. Due to the many scattered and partially similar anatomical structures in standard ultrasound views of the fetal heart, the results strongly depend on the clinical experience and knowledge of the sonographer. To improve detection efficiency and reduce misdiagnosis and omission, we propose a single-stage fetal cardiac ultrasound standard plane detection model (FCUM) based on multitask learning and a hybrid attention mechanism to assist sonographers in diagnosis. The feature fusion pyramids of the backbone and detection networks of this model are each embedded with a hybrid attention mechanism module of our design, which enables the multitasking network to extract shared features more accurately and efficiently and improves the accuracy of the detection and classification networks. We designed a classification module for multilayer residual network feature fusion that leads to better classification and faster convergence time. We conducted comprehensive experiments on a dataset of fetal cardiac ultrasound images acquired with several types of devices and in different geographic regions. The experimental results show that our model outperforms baseline models such as YOLOv8 and ResNet-50 in terms of detection precision and classification accuracy. 

Fetal cardiac ultrasound standard section detection model based on multitask learning and mixed attention mechanism

With the significant advancements of Large Language Models (LLMs) in the field of Natural Language Processing (NLP), the development of image-text multimodal models has garnered widespread attention. Current surveys on image-text multimodal models mainly focus on representative models or application domains, but lack a review on how general technical models influence the development of domain-specific models, which is crucial for domain researchers. Based on this, this paper first reviews the technological evolution of image-text multimodal models, from early explorations of feature space to visual language encoding structures, and then to the latest large model architectures. Next, from the perspective of technological evolution, we explain how the development of general image-text multimodal technologies promotes the progress of multimodal technologies in the biomedical field, as well as the importance and complexity of specific datasets in the biomedical domain. Then, centered on the tasks of image-text multimodal models, we analyze their common components and challenges. After that, we summarize the architecture, components, and data of general image-text multimodal models, and introduce the applications and improvements of image-text multimodal models in the biomedical field. Finally, we categorize the challenges faced in the development and application of general models into external factors and intrinsic factors, further refining them into 2 external factors and 5 intrinsic factors, and propose targeted solutions, providing guidance for future research directions. For more details and data, please visit our GitHub page: https://github.com/i2vec/A-survey-on-image-text-multimodal-models. 

A survey on advancements in image-text multimodal models: From general techniques to biomedical implementations

GraphGuard: Detecting and Counteracting Training Data Misuse in Graph Neural Networks

The utilization of web social media for depression detection has been proven effective in recent years since the multimedia signal on web can reflect users’ emotions, feelings, and personality traits in advance. However, most earlier studies simply used users’ submitted words or user profiles to predict depression risk. The implicit information accessible in users’ posted images, which can be effective in depression detection, still remains unexplored. In this paper, an implicit and explicit multi-modal feature fusion (IEMFF) model is proposed for depression detection. We successfully make the implicit information inherent in users’ posted images explicit and further incorporate such explicit features with the textual features directly extracted from user-posted texts. A multi-modal feature fusion approach is applied for depression detection. Extensive experiments have been conducted on public Twitter datasets. Experimental results show that our approach has achieved state-of-the-art performance for depression detection.

Making the Implicit Explicit: Depression Detection in Web across Posted Texts and Images

Artificial retinas have revolutionized the lives of many blind people by enabling their ability to perceive vision via an implanted chip. Despite significant advancements, there are some limitations that cannot be ignored. Presenting all objects captured in a scene makes their identification difficult. Addressing this limitation is necessary because the artificial retina can utilize a very limited number of pixels to represent vision information. This problem in a multi-object scenario can be mitigated by enhancing images such that only the major objects are considered to be shown in vision. Although simple techniques like edge detection are used, they fall short in representing identifiable objects in complex scenarios, suggesting the idea of integrating primary object edges. To support this idea, the proposed classification model aims at identifying the primary objects based on a suggested set of selective features. The proposed classification model can then be equipped into the artificial retina system for filtering multiple primary objects to enhance vision. The suitability of handling multi-objects enables the system to cope with real-world complex scenarios. The proposed classification model is based on a multi-label deep neural network, specifically designed to leverage from the selective feature set. Initially, the enhanced images proposed in this research are compared with the ones that utilize an edge detection technique for single, dual, and multi-object images. These enhancements are also verified through an intensity profile analysis. Subsequently, the proposed classification model’s performance is evaluated to show the significance of utilizing the suggested features. This includes evaluating the model’s ability to correctly classify the top five, four, three, two, and one object(s), with respective accuracies of up to 84.8%, 85.2%, 86.8%, 91.8%, and 96.4%. Several comparisons such as training/validation loss and accuracies, precision, recall, specificity, and area under a curve indicate reliable results. Based on the overall evaluation of this study, it is concluded that using the suggested set of selective features not only improves the classification model’s performance, but aligns with the specific problem to address the challenge of correctly identifying objects in multi-object scenarios. Therefore, the proposed classification model designed on the basis of selective features is considered to be a very useful tool in supporting the idea of optimizing image enhancement.

Optimizing Image Enhancement: Feature Engineering for Improved Classification in AI-Assisted Artificial Retinas

The abundance of artificial intelligence AI algorithms and growing computing power has brought a disruptive revolution to the smart medical industry. Its powerful data abstraction and representation capabilities enable the modeling of hundreds of millions of medical data, such as sub-Computed Tomography tumor identification, retinal lesion screening, and survival curve analysis. However, all of these applications demonstrate AI’s use of unimodal data for specific tasks. In contrast, clinicians deal with multi-modal data from multiple sources when diagnosing, performing prognostic assessments, and deciding on treatment plans. These requirements have facilitated the development of multi-modal AI solutions and improved the performance of AI models in handling complex medical scenarios and data. In this paper, we provide an overview of the current state of the art and research in multi-modal biomedical AI, including applications, data, methods, and analytics. Additionally, we summarize potential research directions for multi-modal AI technologies in the future of healthcare. 

Artificial intelligence accelerates multi-modal biomedical process: A Survey

Organoid research plays an important role in drug screening and disease modeling. Obtaining accurate information about organoid morphology, number, and size is fundamental to this research. However, previous methods relied on fluorescence labeling which can harm organoids or have problems with accuracy and robustness. In this paper, we first introduce Transformer architecture into the organoid segmentation task and propose an end-to-end multi-modal method named TransOrga. To enhance the accuracy and robustness, we utilize a multi-modal feature extraction module to blend spatial and frequency domain features of organoid images. Furthermore, we propose a multi-branch aggregation decoder that learns diverse contexts from various Transformer layers to predict the segmentation mask progressively. In addition, we design a series of losses, including focal loss, dice loss, compact loss and auxiliary loss, to supervise our model to predict more accurate segmentation results with rational sizes and shapes. Our extensive experiments demonstrate that our method outperforms the baselines in organoid segmentation and provides an automatic, robust, and fluorescent-free tool for organoid research. 

TransOrga: End-To-End Multi-modal Transformer-Based Organoid Segmentation

Autonomous driving systems have been widely adopted by automobile manufacturers, ushering in a new era of intelligent transportation. While adverse weather conditions continue to pose a significant challenge to its commercial application, as they can impact sensor data, degrade the quality of image transmission, and pose safety risks. Using neural network models to remove rain has shown significant promise in addressing this problem. The learning-based rain-removal algorithm discovers the deep connection between rainy pictures and non-rainy pictures by mining the information on raindrops and rain patterns. Nevertheless, the robustness of these rain removal algorithms was not considered, which poses a threat to autonomous vehicles. In this paper, we propose an optimized CW adversarial sample attack to explore the robustness of the rain removal algorithm. In our attacks, we generate a perturbation index of structural similarity that is difficult to detect through human vision and image pixel analysis, causing the similarity and image quality of the restored scene to be significantly degraded. To validate the realistic attack potential of the proposed method, a pre-trained State-of-the-art rain removal attack algorithm, RainCCN, is used as a potential victim of the proposed attack method. We demonstrate the effectiveness of our approach against a state-of-the-art rain removal algorithm, RainCCN, and show that we can reduce PSNR by 39.5 and SSIM by 26.4.

Toward Evaluating the Robustness of Deep Learning Based Rain Removal Algorithm in Autonomous Driving

Momentum contrast [16] (MoCo) for unsupervised visual representation learning has a close performance to supervised learning, but it sometimes possesses excess parameters. Extracting a subnetwork from an over-parameterized unsupervised network without sacrificing performance is of particular interest to accelerate inference speed. Typical pruning methods are not applicable for MoCo, because in the fine-tune stage after pruning, the slow update of the momentum encoder will undermine the pretrained encoder. In this paper, we propose a Momentum Contrastive Pruning (MCP) method, which prunes the momentum encoder instead to obtain a momentum subnet. It maintains an un-pruned momentum encoder as a smooth transition scheme to alleviate the representation gap between the encoder and momentum subnet. To fulfill the sparsity requirements of the encoder, alternating direction method of multipliers [40] (ADMM) is adopted. Experiments prove that our MCP method can obtain a momentum subnet that has almost equal performance as the over-parameterized MoCo when transferred to downstream tasks, meanwhile has much less parameters and float operations per second (FLOPs).

Momentum Contrastive Pruning

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Yiming Qin

Author Tools

Chat about Author

Papers

Artificial intelligence accelerates multi-modal biomedical process: A Survey

TransOrga: End-To-End Multi-modal Transformer-Based Organoid Segmentation

Toward Evaluating the Robustness of Deep Learning Based Rain Removal Algorithm in Autonomous Driving

Momentum Contrastive Pruning