Automatically Optimizing Utterance Classification Performance without Human in the Loop.

TL;DR: A robust recipe for training a UC system using inexpensive acoustic data with limited transcriptions or semantic labels is provided and two new algorithms that use caller confirmation, which naturally occurred within a dialog, to generate pseudo semantic labels are described.

Abstract: The Utterance Classification (UC) method has become a developer’s choice over traditional Context Free Grammars (CFGs) for voice menus in telephony applications. This data driven method achieves higher accuracy and has great potential to utilize a huge amount of labeled training data. But, having a human manually label the training data can be expensive. This paper provides a robust recipe for training a UC system using inexpensive acoustic data with limited transcriptions or semantic labels. It also describes two new algorithms that use caller confirmation, which naturally occurred within a dialog, to generate pseudo semantic labels. Experimental results show that, after having sufficient labeled data to achieve a reasonable accuracy, both of our algorithms can use unlabeled data to achieve the same performance as a system trained with labeled data, while completely eliminating the need for human supervision. Index Terms: Call Routing, Statistical grammars, Spoken language understanding (SLU), Utterance Classification (UC)