Recently, the study of multi-modal brain connectome has recorded a tremendous increase and facilitated the diagnosis of brain disorders. In this paradigm, functional and structural networks, e.g., functional and structural connectivity derived from fMRI and DTI, are in some manner interacted but are not necessarily linearly related. Accordingly, there remains a great challenge to leverage complementary information for brain connectome analysis. Recently, Graph Convolutional Networks (GNN) have been widely applied to the fusion of multi-modal brain connectome. However, most existing GNN methods fail to couple inter-modal relationships. In this regard, we propose a Cross-modal Graph Neural Network (Cross-GNN) that captures inter-modal dependencies through dynamic graph learning and mutual learning. Specifically, the inter-modal representations are attentively coupled into a compositional space for reasoning inter-modal dependencies. Additionally, we investigate mutual learning in explicit and implicit ways: (1) Cross-modal representations are obtained by cross-embedding explicitly based on the inter-modal correspondence matrix. (2) We propose a cross-modal distillation method to implicitly regularize latent representations with cross-modal semantic contexts. We carry out statistical analysis on the attentively learned correspondence matrices to evaluate inter-modal relationships for associating disease biomarkers. Our extensive experiments on three datasets demonstrate the superiority of our proposed method for disease diagnosis with promising prediction performance and multi-modal connectome biomarker location.

Mapping Multi-modal Brain Connectome for Brain Disorder Diagnosis via Cross-modal Mutual Learning.

Structural magnetic resonance imaging (sMRI), which can reflect cerebral atrophy, plays an important role in the early detection of Alzheimer's disease (AD). However, the information provided by analyzing only the morphological changes in sMRI is relatively limited, and the assessment of the atrophy degree is subjective. Therefore, it is meaningful to combine sMRI with other clinical information to acquire complementary diagnosis information and achieve a more accurate classification of AD. Nevertheless, how to fuse these multi-modal data effectively is still challenging. In this paper, we propose DE-JANet, a unified AD classification network that integrates image data sMRI with non-image clinical data, such as age and Mini-Mental State Examination (MMSE) score, for more effective multi-modal analysis. DE-JANet consists of three key components: (1) a dual encoder module for extracting low-level features from the image and non-image data according to specific encoding regularity, (2) a joint attention module for fusing multi-modal features, and (3) a token classification module for performing AD-related classification according to the fused multi-modal features. Our DE-JANet is evaluated on the ADNI dataset, with a mean accuracy of 0.9722 and 0.9538 for AD classification and mild cognition impairment (MCI) classification, respectively, which is superior to existing methods and indicates advanced performance on AD-related diagnosis tasks. 

DE-JANet: A unified network based on dual encoder and joint attention for Alzheimer's disease classification using multi-modal data.

Alzheimer’s disease diagnosis from single and multimodal data using machine and deep learning models: Achievements and future directions

Functional connectivity (FC) networks based on resting-state functional magnetic imaging (rs-fMRI) are reliable and sensitive for brain disorder diagnosis. However, most existing methods are limited by using a single template, which may be insufficient to reveal complex brain connectivities. Furthermore, these methods usually neglect the complementary information between static and dynamic brain networks, and the functional divergence among different brain regions, leading to suboptimal diagnosis performance. To address these limitations, we propose a novel multi-graph cross-attention based region-aware feature fusion network (MGCA-RAFFNet) by using multi-template for brain disorder diagnosis. Specifically, we first employ multi-template to parcellate the brain space into different regions of interest (ROIs). Then, a multi-graph cross-attention network (MGCAN), including static and dynamic graph convolutions, is developed to explore the deep features contained in multi-template data, which can effectively analyze complex interaction patterns of brain networks for each template, and further adopt a dual-view cross-attention (DVCA) to acquire complementary information. Finally, to efficiently fuse multiple static-dynamic features, we design a region-aware feature fusion network (RAFFNet), which is beneficial to improve the feature discrimination by considering the underlying relations among static-dynamic features in different brain regions. Our proposed method is evaluated on both public ADNI-2 and ABIDE-I datasets for diagnosing mild cognitive impairment (MCI) and autism spectrum disorder (ASD). Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art methods. Our source code is available at https://github.com/mylbuaa/MGCA-RAFFNet.

A Multi-Graph Cross-Attention based Region-Aware Feature Fusion Network using Multi-Template for Brain Disorder Diagnosis.

Multi-modal data can provide complementary information of Alzheimer's disease (AD) and its development from different perspectives. Such information is closely related to the diagnosis, prevention, and treatment of AD, and hence it is necessary and critical to study AD through multi-modal data. Existing learning methods, however, usually ignore the influence of feature heterogeneity and directly fuse features in the last stages. Furthermore, most of these methods only focus on local fusion features or global fusion features, neglecting the complementariness of features at different levels and thus not sufficiently leveraging information embedded in multi-modal data. To overcome these shortcomings, we propose a novel framework for AD diagnosis that fuses gene, imaging, protein, and clinical data. Our framework learns feature representations under the same feature space for different modalities through a feature induction learning (FIL) module, thereby alleviating the impact of feature heterogeneity. Furthermore, in our framework, local and global salient multi-modal feature interaction information at different levels is extracted through a novel dual multilevel graph neural network (DMGNN). We extensively validate the proposed method on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset and experimental results demonstrate our method consistently outperforms other state-of-the-art multi-modal fusion methods. The code is publicly available on the GitHub website. (https://github.com/xiankantingqianxue/MIA-code.git) 

Alzheimer’s disease diagnosis from multi-modal data via feature inductive learning and dual multilevel graph neural network

Multi-modal fusion has become an important data analysis technology in Alzheimer’s disease (AD) diagnosis, which is committed to effectively extract and utilize complementary information among different modalities. However, most of the existing fusion methods focus on pursuing common feature representation by transformation, and ignore discriminative structural information among samples. In addition, most fusion methods use high-order feature extraction, such as deep neural network, by which it is difficult to identify biomarkers. In this paper, we propose a novel method named deep multi-modal discriminative and interpretability network (DMDIN), which aligns samples in a discriminative common space and provides a new approach to identify significant brain regions (ROIs) in AD diagnosis. Specifically, we reconstruct each modality with a hierarchical representation through multilayer perceptron (MLP), and take advantage of the shared self-expression coefficients constrained by diagonal blocks to embed the structural information of inter-class and the intra-class. Further, the generalized canonical correlation analysis (GCCA) is adopted as a correlation constraint to generate a discriminative common space, in which samples of the same category gather while samples of different categories stay away. Finally, in order to enhance the interpretability of the deep learning model, we utilize knowledge distillation to reproduce coordinated representations and capture influence of brain regions in AD classification. Experiments show that the proposed method performs better than several state-of-the-art methods in AD diagnosis.

Deep Multi-Modal Discriminative and Interpretability Network for Alzheimer’s Disease Diagnosis

Multi-modal imaging data fusion has attracted much attention in medical data analysis because it can provide complementary information for more accurate analysis. Integrating functional and structural multi-modal imaging data has been increasingly used in the diagnosis of brain diseases, such as epilepsy. Most of the existing methods focus on the feature space fusion of different modalities but ignore the valuable high-order relationships among samples and the discriminative fused features for classification. In this paper, we propose a novel framework by fusing data from two modalities of functional MRI (fMRI) and diffusion tensor imaging (DTI) for epilepsy diagnosis, which effectively captures the complementary information and discriminative features from different modalities by high-order feature extraction with the attention mechanism. Specifically, we propose a triple network to explore the discriminative information from the high-order representation feature space learned from multi-modal data. Meanwhile, self-attention is introduced to adaptively estimate the degree of importance between brain regions, and the cross-attention mechanism is utilized to extract complementary information from fMRI and DTI. Finally, we use the triple loss function to adjust the distance between samples in the common representation space. We evaluate the proposed method on the epilepsy dataset collected from Jinling Hospital, and the experiment results demonstrate that our method is significantly superior to several state-of-the-art diagnosis approaches.

Multimodal Triplet Attention Network for Brain Disease Diagnosis

This retrospective study aimed to investigate the clinical characteristics, prognosis, and treatment outcomes of liver metastatic cancer in patients with colorectal cancer. Medical records of patients diagnosed with colorectal cancer and concurrent hepatic metastatic cancer were collected from our hospital between October 2016 and June 2019. Among the 33 included patients, liver metastatic cancer accounted for 22.7% of the total diagnosed cases. The male-to-female ratio was 58:42, and the mean age was 55 years (range: 45-75 years). The liver metastatic carcinoma exhibited the same pathological type as the primary tumor, specifically adenosquamous cell tumor. Clinical manifestations of liver metastatic cancer mainly included symptoms such as ascites, splenomegaly, and impaired liver function. The prognosis of patients with liver metastatic cancer was found to be poor, with a 5-year survival rate of approximately 30%. These findings highlight the high incidence of liver metastatic cancer in colorectal cancer patients and the worse prognosis compared to the primary tumor. It emphasizes the importance of regular liver examinations for colorectal cancer patients to facilitate early detection of liver metastases and the implementation of effective treatment strategies to improve patient survival rates.

Analysis of Treatment and Survival Factors of Colorectal Cancer with Multiple Liver Metastases

At present, most of the data-driven image medical analysis methods in the diagnosis of brain diseases are difficult to utilize the existing medical expert knowledge, which is unconducive to improve the robustness of model. The cognition scale naturally contains expert knowledge of related brain diseases, but the hybrid-driven method fusing brain imaging and cognition scales to predict brain diseases has challenges in data heterogeneity and information interaction among different analysis tasks. To address the problem above, we propose a contrastive multitask self-supervised pretraining model by effectively fusing brain imaging and cognition scales. First, we simultaneously establish the brain image feature reconstruction and the cognitive score regression prediction model in uniform framework, which can extract the complementary information between brain imaging feature and cognitive scale. Second, we introduce the mask automatic encoder, a self-supervised learning method, to above model to make full use of the rich feature information of sMRI. Then, we adopt contrastive learning to alleviate the problem of negative transfer between the brain reconstruction and cognition score regression prediction tasks. Finally, the loss functions of the two tasks are jointly used to train the backbone network alternately, and the model is applied to the downstream AD diagnosis task as a pre-training model. The experimental results demonstrate that our proposed method is superior to other common AD diagnosis methods, and the proposed model has a certain generalization ability.

Hybrid Intelligent Model Fusing Brain Imaging and Cognition Scale Information for AD Diagnosis

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Heyang Wang

Author Tools

Chat about Author

Papers

Deep Multi-Modal Discriminative and Interpretability Network for Alzheimer’s Disease Diagnosis

Multimodal Triplet Attention Network for Brain Disease Diagnosis

Analysis of Treatment and Survival Factors of Colorectal Cancer with Multiple Liver Metastases

Hybrid Intelligent Model Fusing Brain Imaging and Cognition Scale Information for AD Diagnosis