Recurrent Neural Networks: A Comprehensive Review of Architectures, Variants, and Applications

Physical reservoir computing with emerging electronics

This paper targets at improving the generalizability of hypergraph neural networks in the low-label regime, through applying the contrastive learning approach from images/graphs (we refer to it as HyperGCL). We focus on the following question: How to construct contrastive views for hypergraphs via augmentations? We provide the solutions in two folds. First, guided by domain knowledge, we fabricate two schemes to augment hyperedges with higher-order relations encoded, and adopt three vertex augmentation strategies from graph-structured data. Second, in search of more effective views in a data-driven manner, we for the first time propose a hypergraph generative model to generate augmented views, and then an end-to-end differentiable pipeline to jointly learn hypergraph augmentations and model parameters. Our technical innovations are reflected in designing both fabricated and generative augmentations of hypergraphs. The experimental findings include: (i) Among fabricated augmentations in HyperGCL, augmenting hyperedges provides the most numerical gains, implying that higher-order information in structures is usually more downstream-relevant; (ii) Generative augmentations do better in preserving higher-order information to further benefit generalizability; (iii) HyperGCL also boosts robustness and fairness in hypergraph representation learning. Codes are released at https://github.com/weitianxin/HyperGCL.

pdf/augmentations-in-hypergraph-contrastive-learning-fabricated-8rq9jnvd.pdf

Augmentations in Hypergraph Contrastive Learning: Fabricated and Generative

Electronic Health Record modeling is crucial for digital medicine. However, existing models ignore higher-order interactions among medical codes and their causal relations towards downstream clinical predictions. To address such limitations, we propose a novel framework CACHE, to provide effective and insightful clinical predictions based on hypergraph representation learning and counterfactual and factual reasoning techniques. Experiments on two real EHR datasets show the superior performance of CACHE. Case studies with a domain expert illustrate a preferred capability of CACHE in generating clinically meaningful interpretations towards the correct predictions.

Counterfactual and Factual Reasoning over Hypergraphs for Interpretable Clinical Predictions on EHR

Plenty of models have been presented to handle the hypergraph node classification. However, very few of these methods consider contrastive learning, which is popular due to its great power to represent instances. This paper makes an attempt to leverage contrastive learning to hypergraph representation learning. Specifically, we propose a novel method called Collaborative Contrastive Learning (CCL), which incorporates a generated standard graph with the hypergraph. The main technical contribution here is that we develop a collaborative contrastive schema, which performs contrast between the node views obtained from the standard graph and hypergraph in each network layer, thus making the contrast collaborative. To be precise, in the first layer, the view from the standard graph is used to augment that from the hypergraph. Then, in the next layer, the augmented features are adopted to train a new representation to augment the view from the standard graph conversely. With this setting, the learning procedure is alternated between the standard graph and hypergraph. As a result, the learning on the standard graph and hypergraph is collaborative and leads to the final informative node representation. Experimental results on several widely used datasets validate the effectiveness of the proposed model. 

Collaborative contrastive learning for hypergraph node classification

Hypergraphs are natural and expressive modeling tools to encode high-order relationships among entities. Several variations of Hypergraph Neural Networks (HGNNs) are proposed to learn the node representations and complex relationships in the hypergraphs. Most current approaches assume that the input hypergraph structure accurately depicts the relations in the hypergraphs. However, the input hypergraph structure inevitably contains noise, task-irrelevant information, or false-negative connections.

Treating the input hypergraph structure as ground-truth information unavoidably leads to sub-optimal performance. In this paper, we propose a Hypergraph Structure Learning (HSL) framework, which optimizes the hypergraph structure and the HGNNs simultaneously in an end-to-end way. 

HSL learns an informative and concise hypergraph structure that is optimized for downstream tasks. To efficiently learn the hypergraph structure, HSL adopts a two-stage sampling process: hyperedge sampling for pruning redundant hyperedges and incident node sampling for pruning irrelevant incident nodes and discovering potential implicit connections. 

The consistency between the optimized structure and the original structure is maintained by the intra-hyperedge contrastive learning module.

The sampling processes are jointly optimized with HGNNs towards the objective of the downstream tasks. Experiments conducted on 7 datasets show shat HSL outperforms the state-of-the-art baselines while adaptively sparsifying hypergraph structures.

Hypergraph Structure Learning for Hypergraph Neural Networks

Recurrent Neural Network (RNN) has demonstrated its outstanding ability in sequence tasks and has achieved state-of-the-art in many applications, such as industrial and medical. Echo State Network (ESN) is a simple type of RNN and has emerged in the last decade as an alternative to gradient descent training-based RNN. ESN is practical, conceptually simple, and easy to implement with a strong theoretical ground. It can avoid non-converging and computationally expensive issues in gradient descent RNN methods. Since ESN was put forward in 2002, abundant existing works have promoted the progress of ESN, and the recently introduced deep ESN opened the way to uniting the merits of deep learning and reservoir computing. Besides, the combinations of ESNs with other machine learning models have also overperformed baselines in some applications. However, the apparent simplicity of ESNs can sometimes be deceptive. Successfully applying ESNs needs some experience. Thus, we reviewed over 300 related papers and provided a systematic overview for the first time. In this paper, we categorize the related methods into classical ESN, DeepESN, and combination. Then, we analyze them from the perspective of network designs and specific applications. Finally, we discuss the challenges and opportunities by proposing open problems and future work. 

A Systematic Review of Echo State Networks from Design to Application

Irregularly Sampled Time Series (ISTS) include partially observed feature vectors caused by the lack of temporal alignment across dimensions and the presence of variable time intervals. Especially in medical applications, because patients' examinations depend on their health status, observations in this event-based medical time series are nonuniformly distributed. When using deep learning models to classify ISTS, most work defines the problem that needs to be solved as alignment-caused data missing or nonuniformity-caused dependency change. However, they only modeled relationships between observed values, ignoring the fact that time is the independent variable for a time series. In this paper, we emphasize that irregularity is active, time-depended, and class-associated and is reflected in the Time Pattern (TP). To this end, this paper focused on the TP of ISTS for the first time, proposing a Time Pattern Reconstruction (TPR) method. It first encodes time information by the time encoding mechanism, then imputes values from time codes by the continuous-discrete Kalman network, selects key time points by the conditional masking mechanism, and finally classifies ISTS based on the reconstructed TP. Experiments on four real-world medical datasets and three other datasets show that TPR outperforms all baselines. We also show that TP can reveal biomarkers and key time points for diseases. 

Time pattern reconstruction for classification of irregularly sampled time series

In the real world, the class of a time series is usually labeled at the final time, but many applications require to classify time series at every time point. e.g. the outcome of a critical patient is only determined at the end, but he should be diagnosed at all times for timely treatment. Thus, we propose a new concept: Continuous Classification of Time Series (CCTS). It requires the model to learn data in different time stages. But the time series evolves dynamically, leading to different data distributions. When a model learns multi-distribution, it always forgets or overfits. We suggest that meaningful learning scheduling is potential due to an interesting observation: Measured by confidence, the process of model learning multiple distributions is similar to the process of human learning multiple knowledge. Thus, we propose a novel Confidence-guided method for CCTS (C3TS). It can imitate the alternating human confidence described by the Dunning-Kruger Effect. We define the objective-confidence to arrange data, and the self-confidence to control the learning duration. Experiments on four real-world datasets show that C3TS is more accurate than all baselines for CCTS.

/pdf/confidence-guided-learning-process-for-continuous-189cssqz.pdf

Confidence-Guided Learning Process for Continuous Classification of Time Series

Length of Stay (LoS) estimation is important for efficient healthcare resource management. Since the distribution of LoS is highly skewed, some previous works frame the LoS estimation as a multi-class classification problem by dividing the range of LoS into buckets. However, they ignore the ordinal relationship between labels. The distribution of bucketed LoS, with a heavy head and a heavy tail, is still imbalanced since the long tail is grouped into the last bucket. This paper proposes a Deep Ordinal neural network for Length of stay Estimation in the intensive care units (DOSE). DOSE can exploit the ordinal relationship and mitigate the skewness. The ordinal classification problem is decomposed into a series of binary classification sub-problems by using multiple binary classifiers. To maintain consistency among binary classifiers, the monotonicity constraint penalty is proposed. The number of samples whose labels are higher or lower than a given threshold is at the same level due to the heavy head and tail of the distribution. Therefore, the training data of each binary classifier are balanced. Experiments are conducted on the real-world healthcare dataset. DOSE outperforms all baseline methods in all metrics. The distribution of the prediction of DOSE is more aligned with the ground truth.

Deep Ordinal Neural Network for Length of Stay Estimation in the Intensive Care Units

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

D. Cai

Author Tools

Chat about Author