BERT (Bidirectional Encoder Representations from Transformers) for Missing Data Imputation in Solar Irradiance Time Series

Abstract: Transformer models have emerged as pivotal tools within the realm of drug discovery, distinguished by their unique architectural features and exceptional performance in managing intricate data landscapes. Leveraging the innate capabilities of transformer architectures to comprehend intricate hierarchical dependencies inherent in sequential data, these models showcase remarkable efficacy across various tasks, including new drug design and drug target identification. The adaptability of pre-trained transformer-based models renders them indispensable assets for driving data-centric advancements in drug discovery, chemistry, and biology, furnishing a robust framework that expedites innovation and discovery within these domains. Beyond their technical prowess, the success of transformer-based models in drug discovery, chemistry, and biology extends to their interdisciplinary potential, seamlessly combining biological, physical, chemical, and pharmacological insights to bridge gaps across diverse disciplines. This integrative approach not only enhances the depth and breadth of research endeavors but also fosters synergistic collaborations and exchange of ideas among disparate fields. In our review, we elucidate the myriad applications of transformers in drug discovery, as well as chemistry and biology, spanning from protein design and protein engineering, to molecular dynamics (MD), drug target identification, transformer-enabled drug virtual screening (VS), drug lead optimization, drug addiction, small data set challenges, chemical and biological image analysis, chemical language understanding, and single cell data. Finally, we conclude the survey by deliberating on promising trends in transformer models within the context of drug discovery and other sciences.

TL;DR: This study combines machine learning and deep learning models with natural language processing to enhance autism spectrum disorder (ASD) detection accuracy from social media text inputs, achieving an 88% success rate in identifying ASD texts.

TL;DR: The first version of a computational solution was built to consume the language models based on BERT and Llama 2-Chat and thus automatically interpret pre-conceptual schemas by computers via natural language processing, adding, at the same time, generative capabilities.

TL;DR: Deep learning is revolutionizing structural bioinformatics, enabling advancements in biomolecular structure understanding and revolutionizing analytical processes.

Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

TL;DR: BEVT as discussed by the authors decouples video representation learning into spatial representation learning and temporal dynamics learning and achieves state-of-the-art performance on three challenging video benchmarks, including Kinetics 400, Diving 48, and Something-Something-V2.

TL;DR: In this paper, the estimation of missing values in minutely, hourly, daily, and weekly solar irradiance series using an extensive number of imputation methods was investigated, including linear and Stineman interpolations and Kalman filtering with structural model and smoothing.

TL;DR: A novel solarGAN method for multivariate solar data imputation is proposed, in which necessary modifications are made on the input of generative adversarial network (GAN) to effectively tackle the relatively independent solar time series data.