Audio pattern recognition is an important research topic in the machine learning area, and includes several tasks such as audio tagging, acoustic scene classification, music classification, speech emotion classification and sound event detection. Recently, neural networks have been applied to tackle audio pattern recognition problems. However, previous systems are built on specific datasets with limited durations. Recently, in computer vision and natural language processing, systems pretrained on large-scale datasets have generalized well to several tasks. However, there is limited research on pretraining systems on large-scale datasets for audio pattern recognition. In this paper, we propose pretrained audio neural networks (PANNs) trained on the large-scale AudioSet dataset. These PANNs are transferred to other audio related tasks. We investigate the performance and computational complexity of PANNs modeled by a variety of convolutional neural networks. We propose an architecture called Wavegram-Logmel-CNN using both log-mel spectrogram and waveform as input feature. Our best PANN system achieves a state-of-the-art mean average precision (mAP) of 0.439 on AudioSet tagging, outperforming the best previous system of 0.392. We transfer PANNs to six audio pattern recognition tasks, and demonstrate state-of-the-art performance in several of those tasks. We have released the source code and pretrained models of PANNs: https://github.com/qiuqiangkong/audioset_tagging_cnn .

pdf/panns-large-scale-pretrained-audio-neural-networks-for-audio-4vxeip6al9.pdf

PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

Earthquake signal detection and seismic phase picking are challenging tasks in the processing of noisy data and the monitoring of microearthquakes. Here we present a global deep-learning model for simultaneous earthquake detection and phase picking. Performing these two related tasks in tandem improves model performance in each individual task by combining information in phases and in the full waveform of earthquake signals by using a hierarchical attention mechanism. We show that our model outperforms previous deep-learning and traditional phase-picking and detection algorithms. Applying our model to 5 weeks of continuous data recorded during 2000 Tottori earthquakes in Japan, we were able to detect and locate two times more earthquakes using only a portion (less than 1/3) of seismic stations. Our model picks P and S phases with precision close to manual picks by human analysts; however, its high efficiency and higher sensitivity can result in detecting and characterizing more and smaller events. The authors here present a deep learning model that simultaneously detects earthquake signals and measures seismic-phase arrival times. The model performs particularly well for cases with high background noise and the challenging task of picking the S wave arrival.

/pdf/earthquake-transformer-an-attentive-deep-learning-model-for-24zhtci4pb.pdf

Earthquake transformer-an attentive deep-learning model for simultaneous earthquake detection and phase picking.

Automatic sleep stage mymargin classification is of great importance to measure sleep quality. In this paper, we propose a novel attention-based deep learning architecture called AttnSleep to classify sleep stages using single channel EEG signals. This architecture starts with the feature extraction module based on multi-resolution convolutional neural network (MRCNN) and adaptive feature recalibration (AFR). The MRCNN can extract low and high frequency features and the AFR is able to improve the quality of the extracted features by modeling the inter-dependencies between the features. The second module is the temporal context encoder (TCE) that leverages a multi-head attention mechanism to capture the temporal dependencies among the extracted features. Particularly, the multi-head attention deploys causal convolutions to model the temporal relations in the input features. We evaluate the performance of our proposed AttnSleep model using three public datasets. The results show that our AttnSleep outperforms state-of-the-art techniques in terms of different evaluation metrics. Our source codes, experimental data, and supplementary materials are available at https://github.com/emadeldeen24/AttnSleep .

/pdf/an-attention-based-deep-learning-approach-for-sleep-stage-2dy2yrt1et.pdf

An Attention-Based Deep Learning Approach for Sleep Stage Classification With Single-Channel EEG

Speech-driven facial animation is the process that automatically synthesizes talking characters based on speech signals. The majority of work in this domain creates a mapping from audio features to visual features. This approach often requires post-processing using computer graphics techniques to produce realistic albeit subject dependent results. We present an end-to-end system that generates videos of a talking head, using only a still image of a person and an audio clip containing speech, without relying on handcrafted intermediate features. Our method generates videos which have (a) lip movements that are in sync with the audio and (b) natural facial expressions such as blinks and eyebrow movements. Our temporal GAN uses 3 discriminators focused on achieving detailed frames, audio-visual synchronization, and realistic expressions. We quantify the contribution of each component in our model using an ablation study and we provide insights into the latent representation of the model. The generated videos are evaluated based on sharpness, reconstruction quality, lip-reading accuracy, synchronization as well as their ability to generate natural blinks.

/pdf/realistic-speech-driven-facial-animation-with-gans-4wfu2282gn.pdf

Realistic Speech-Driven Facial Animation with GANs

Environmental sound classification task (ESC) is still open and challenging. In contrast to speech, sounds of a specific acoustic event may be produced by a wide variety of sources. Thus for one class, feature spectrums of acoustic events are much more transformative than human speech. In order to learn better high-level feature representations from these transformative feature spectrums, convolution neural network (CNN) has been applied to ESC tasks and achieved state-of-the-art results. However, it is noted that existing CNN-based ESC systems only use small convolution filters (typically 2×2 or 3×3) in CNN model which results in deep CNN model for learning long contextual information of sound events. Besides, to our knowledge, there is no work of evaluating the effect of activation functions on the performance of CNN-based ESC systems, which is actually very critical for improving the performance. Motivated by these findings, in this study, we propose a dilated CNN-based ESC (D-CNN-ESC) system where dilated filters and LeakyReLU activation function are adopted. The main ideas behind our research are that the dilated filters will increase receptive field of convolution layers to incorporate more contextual information. Moreover, the LeakyReLU function brings the tradeoff between the network sparsity and the performance. To evaluate the effectiveness of our proposed D-CNN-ESC system, we conduct several experiments on three sound event datasets. It is encouraged to see that our proposed D-CNN-ESC system outperforms the state-of-the-art ESC results obtained by very deep CNN-ESC system on UrbanSound8K dataset, the absolute error of our method is about 10% less than that of compared method.

Dilated convolution neural network with LeakyReLU for environmental sound classification

Learning acoustic models directly from the raw waveform data with minimal processing is challenging. Current waveform-based models have generally used very few (∼2) convolutional layers, which might be insufficient for building high-level discriminative features. In this work, we propose very deep convolutional neural networks (CNNs) that directly use time-domain waveforms as inputs. Our CNNs, with up to 34 weight layers, are efficient to optimize over very long sequences (e.g., vector of size 32000), necessary for processing acoustic waveforms. This is achieved through batch normalization, residual learning, and a careful design of down-sampling in the initial layers. Our networks are fully convolutional, without the use of fully connected layers and dropout, to maximize representation learning. We use a large receptive field in the first convolutional layer to mimic bandpass filters, but very small receptive fields subsequently to control the model capacity. We demonstrate the performance gains with the deeper models. Our evaluation shows that the CNN with 18 weight layers outperforms the CNN with 3 weight layers by over 15% in absolute accuracy for an environmental sound recognition task and is competitive with the performance of models using log-mel features.

pdf/very-deep-convolutional-neural-networks-for-raw-waveforms-2fjjznu9iy.pdf

Very deep convolutional neural networks for raw waveforms

Learning acoustic models directly from the raw waveform data with minimal processing is challenging. Current waveform-based models have generally used very few (~2) convolutional layers, which might be insufficient for building high-level discriminative features. In this work, we propose very deep convolutional neural networks (CNNs) that directly use time-domain waveforms as inputs. Our CNNs, with up to 34 weight layers, are efficient to optimize over very long sequences (e.g., vector of size 32000), necessary for processing acoustic waveforms. This is achieved through batch normalization, residual learning, and a careful design of down-sampling in the initial layers. Our networks are fully convolutional, without the use of fully connected layers and dropout, to maximize representation learning. We use a large receptive field in the first convolutional layer to mimic bandpass filters, but very small receptive fields subsequently to control the model capacity. We demonstrate the performance gains with the deeper models. Our evaluation shows that the CNN with 18 weight layers outperform the CNN with 3 weight layers by over 15% in absolute accuracy for an environmental sound recognition task and matches the performance of models using log-mel features.

Very Deep Convolutional Neural Networks for Raw Waveforms

This study aims to improve the source separation technique, DEMUCS, by revising the respective loss function. DEMUCS, developed by Facebook Team, is built on the Wave-U-Net and consists of convolutional layer encoding and decoding blocks with an LSTM layer in between. The applied loss function in DEMUCS contains wave-domain L1 distance and multi-scale short-time-Fourier-transform (STFT) loss.We present to revise the original loss by considering the perceptual metric scores, including perceptual speech quality (PESQ) and short-time objective intelligibility (STOI). The new loss function becomes a weighted sum of the original loss and the losses of STOI and PESQ, hoping to highlight the perceptual quality of the enhanced utterances.According to the preliminary experiments conducted on the VoiceBank-DEMUCS task, the DEMUCS network with the modified loss function provides the noise-corrupted utterances with superior objective perceptual metric scores (PESQ and STOI). These results indicate that the presented work benefits DEMUCS in speech enhancement performance.

Leveraging the perceptual metric loss to improve the DEMUCS system in speech enhancement

This study aims to improve a highly effective speech enhancement technique, DEMUCS, by revising the respective loss function in learning. DEMUCS, developed by Facebook Team, is built on the Wave-UNet and consists of convolutional layer encoding and decoding blocks with an LSTM layer in between. Although DEMUCS processes the input speech utterance purely in the time (wave) domain, the applied loss function consists of wave-domain L1 distance and multi-scale shorttime-Fourier-transform (STFT) loss. That is, both time- and frequency-domain features are taken into consideration in the learning of DEMUCS. In this study, we present revising the STFT loss in DEMUCS by employing the compressed magnitude spectrogram. The compression is done by either the power-law operation with a positive exponent less than one, or the logarithmic operation. We evaluate the presented novel framework on the VoiceBank-DEMAND database and task. The preliminary experimental results suggest that DEMUCS containing the power-law compressed magnitude spectral loss outperforms the original DEMUCS by providing the test utterances with higher objective quality and intelligibility scores (PESQ and STOI). Relatively, the logarithm compressed magnitude spectral loss does not benefit DEMUCS. Therefore, we reveal that DEMUCS can be further improved by properly revising the STFT terms of its loss function.

pdf/exploiting-the-compressed-spectral-loss-for-the-learning-of-2yehlefs.pdf

Exploiting the compressed spectral loss for the learning of the DEMUCS speech enhancement network

This study aims to promote the usage of a renowned speech enhancement method, namely the Conformer-based Metric Generative Adversarial Network (CMGAN). The CMGAN model utilizes a generative adversarial network (GAN) architecture and incorporates a Conformer- based metric to assess the quality of the synthetic data from the generator network. The generator network in CMGAN derives its loss function primarily from three distinct sources, namely: the time-frequency domain loss, the waveform-domain loss, and the loss fed by the discriminator network. The present study posits that the loss function can be subject to further revision through the incorporation of the phone- fortified perceptual loss (PFPL). PFPL is observed as the difference between the latent phone-level representation generated by the wave2vec encoder for synthetic data and that for real data. The preliminary experiments carried out on the VoiceBank-DEMAND task suggest that the incorporation of the PFPL into the CMGAN learning process leads to a notable enhancement in the objective metrics (PESQ and STOI) scores for the test data. 

Improving the performance of CMGAN in speech enhancement with the phone fortified perceptual loss

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Chia Dai

Author Tools

Chat about Author