Spoken keyword spotting (KWS) deals with the identification of keywords in audio streams and has become a fast-growing technology thanks to the paradigm shift introduced by deep learning a few years ago. This has allowed the rapid embedding of deep KWS in a myriad of small electronic devices with different purposes like the activation of voice assistants. Prospects suggest a sustained growth in terms of social use of this technology. Thus, it is not surprising that deep KWS has become a hot research topic among speech scientists, who constantly look for KWS performance improvement and computational complexity reduction. This context motivates this paper, in which we conduct a literature review into deep spoken KWS to assist practitioners and researchers who are interested in this technology. Specifically, this overview has a comprehensive nature by covering a thorough analysis of deep KWS systems (which includes speech features, acoustic modeling and posterior handling), robustness methods, applications, datasets, evaluation metrics, performance of deep KWS systems and audio-visual KWS. The analysis performed in this paper allows us to identify a number of directions for future research, including directions adopted from automatic speech recognition research and directions that are unique to the problem of spoken KWS. 

Deep Spoken Keyword Spotting: An Overview

Even though attention-based end-to-end (E2E) automatic speech recognition (ASR) models have been yielding state-of-the-art recognition accuracy, they still fall behind many of the ASR models deployed in the industry in some crucial functionalities such as online processing and precise timestamps generating. This weakness prevents attention-based E2E ASR models from being applied in several essential speech tasks, such as online speech recognition and keyword searching (KWS). In this paper, we describe our proposed unified attention-based E2E ASR and KWS architecture–ETEH, which supports, in one model, both online and offline ASR decoding modes, thus allowing for precise and reliable KWS. “ETE” stands for attention-based E2E modeling, whereas “H” represents the hybrid gaussian mixture model and hidden Markov model (GMM-HMM). As a combination of both, ETEH is an attention-based E2E ASR architecture which utilizes the frame-wise time alignment (FTA) generated by GMM-HMM ASR models. This FTA is used to better the model in two ways: first, it helps the monotonic attentions of ETEH models to capture more accurate word time stamps, thus resulting in lower latency for online decoding; second, it helps ETEH models to provide accurate and reliable KWS results. Furthermore, we are able to combine both offline and online modes in one ETEH model and establish a concise system by adopt the universal training strategy. ETEH is functional and unique, and to the best of our knowledge, we can hardly find a comparable single attention-based E2E ASR system as the baseline. To evaluate ASR accuracy and latency for ETEH, we use our previously proposed monotonic truncated attention (MTA) based online CTC/attention (OCA) ASR models as baselines. Experimental results show that ETEH ASR models gain significant improvement in ASR latency compared to the baseline. To evaluate KWS performance, we compare ETEH models with CTC-based KWS models. Results demonstrate that our ETEH models achieve significantly better KWS performance compared to the CTC baselines.

ETEH: Unified Attention-Based End-to-End ASR and KWS Architecture

Attention-based end-to-end (E2E) automatic speech recognition (ASR) architectures are now the state-of-the-art in terms of recognition performance. However, despite their effectiveness, they have not been widely applied in keyword search (KWS) tasks yet. In this paper, we propose the Att-E2E-KWS architecture, an attention-based E2E ASR framework for KWS that can afford accurate and reliable keyword retrieval results. First, we design a basic framework to carry out KWS based on attention-based E2E ASR. We adopt the connectionist temporal classification and attention (CTC/Att) joint E2E ASR architecture and exploit the spike posterior property of CTC to provide the keywords time stamps. Second, we introduce the frame-synchronous phonemes modeling and use the dynamic programming (DP) algorithm to provide alignments between E2E grapheme outputs and phoneme outputs. We call this alignment procedure dynamic time alignment (DTA), which can provide the proposed Att-E2E-KWS system with more accurate time stamps and reliable confidence scores. Third, we use the Transformer, a self-attention-based encoder-decoder neural network, in place of conventional recurrent neural networks in order to yield more parallelizable models and increased training speed. We conduct comprehensive experiments on English and Mandarin Chinese. To the best of our knowledge, this is the first practical Att-E2E-KWS framework, and experimental results on Switchboard and HKUST corpora show that our proposed Att-E2E-KWS systems significantly outperform the CTC E2E ASR based KWS baselines.

Keyword Search Using Attention-Based End-to-End ASR and Frame-Synchronous Phoneme Alignments

In the future manned lunar exploration mission, astronauts would work with the lunar robots, which has a high requirement for human–robot interaction (HRI). As the accuracy of gesture recognition interaction does not fulfill the requirement for human–robot joint exploration missions, we propose the DMS-SK/BLSTM-CTC hybrid network to improve the performance of HRI. For gesture recognition, considering VGG-SK has low accuracy and complex architecture, we delete the fourth convolution module, optimize the last global pooling layer, introduce dilated convolution block and multiscale convolution block in VGG-SK, and get the DMS-SK-based gesture recognition sub-network. Compared with the traditional recognition methods, the accuracy and performance of DMS-SK improve. For speech recognition, considering that Bidirectional long–short-term memory unit (BLSTM) has the advantages of processing temporal information, and the Connectionist Temporal Classification (CTC) algorithm can simplify speech data preprocessing, we use BLSTM based on CTC as the speech recognition sub-network. Finally, we combine DMS-SK with BLSTM-CTC, and propose the DMS-SK/BLSTM-CTC hybrid network as the gesture/speech hybrid network. In addition, we use 10 gestures in the American Sign Language (ASL) dataset and 10 speech commands to construct the gesture/speech hybrid dataset. Experimental results show that compared with the pure gesture or pure speech networks, the recognition accuracy of the gesture-speech hybrid network improves by 2% and 12%, respectively, its accuracy reaches 97.38%, which fulfills the requirement of astronauts for HRI.

DMS-SK/BLSTM-CTC Hybrid Network for Gesture/Speech Fusion and Its Application in Lunar Robot-Astronauts Interaction

As more and more information-rich data like video become available, utilizing multi-modal auxiliary information to enhance audio tasks has sparked widespread research interest. The recent surge in research on LLM-based audio models provides fresh perspectives for tackling audio tasks. Given that LLM can flexibly ingest multiple inputs, we propose MaLa-ASR, an LLM-based ASR model that can integrate textual keywords extracted from presentation slides to improve recognition of conference content. MaLa-ASR yields average WERs of 9.4% and 11.7% on the L95 and S95 subsets of the SlideSpeech corpus, representing a significant relative WER drop of 27.9% and 44.7% over the baseline model reported in SlideSpeech. MaLa-ASR underscores LLM's strong performance in speech tasks and the capability to integrate auxiliary information conveniently. By adding keywords to the input prompt, the biased word error rate (B-WER) reduces relatively by 46.0% and 44.2%, establishing a new SOTA on this dataset. 

pdf/mala-asr-multimedia-assisted-llm-based-asr-ekleo9gmc4.pdf

MaLa-ASR: Multimedia-Assisted LLM-Based ASR

The state-of-the-art acoustic modeling for Keyword Spotting (KWS) systems is mainly based on the hybrid model of Hidden Markov Model (HMM) and Neural Network (NN). However, it is challenging to efficiently train such a hybrid system, since the dependence of the intermediate phonetic representation. Motivated by the end-to-end speech recognition systems, we propose a Mandarin KWS system using the end-to-end method, which directly predict the posterior of phonetic units. The system is based on Connectionist Temporal Classifier (CTC) and Recurrent Neural Network (RNN). The main difference between our system and other CTC-based KWS system is the output alphabet and its corresponded keyword searching mechanism. We adopt Mandarin syllables as the output labels, rather than the phonemes or characters. Extensive experiments are conducted on the Mandarin Chinese speech dataset. Experimental results indicate that: 1) Compared with HMM-based KWS system, the end-to-end KWS system achieves a significant improvement, without any increase of computational cost. 2) Our syllable-based end-to-end KWS system obtains better performance than the state-of-the-art ones based on Chinese context independent (CI) phonemes or Chinese characters.

Keyword Spotting Based On CTC and RNN For Mandarin Chinese Speech

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Yiyan Wang

Author Tools

Chat about Author

Papers

Keyword Spotting Based On CTC and RNN For Mandarin Chinese Speech