To support applications of speech-driven interactive systems in various conversational scenarios, text-to-speech (TTS) synthesis needs to understand the conversational context and determine appropriate speaking styles in its synthesized speeches. These speaking styles are influenced by the dependencies between the multi-modal information in the context at both global scale (i.e. utterance level) and local scale (i.e. word level). However, the dependency modeling and speaking style inference at the local scale are largely missing in state-of-the-art TTS systems, resulting in the synthesis of incorrect or improper speaking styles. In this paper, to learn the dependencies in conversations at both global and local scales and to improve the synthesis of speaking styles, we propose a context modeling method which models the dependencies among the multi-modal information in context with multi-scale relational graph convolutional network (MSRGCN). The learnt multi-modal context information at multiple scales is then utilized to infer the global and local speaking styles of the current utterance for speech synthesis. Experiments demonstrate the effectiveness of the proposed approach, and ablation studies reflect the contributions from modeling multi-modal information and multi-scale dependencies.

/pdf/inferring-speaking-styles-from-multi-modal-conversational-ix0s6l3p.pdf

Inferring Speaking Styles from Multi-modal Conversational Context by Multi-scale Relational Graph Convolutional Networks

The aim of emotion prediction in conversation (EPC) is to predict the future emotional state of a speaker based on context information, which is essential for conducting a friendly human-computer conversation. Most EPC works only investigated context information by merging a speaker’s multiple utterances into a single utterance per turn and focused on conversations in a dual-speaker scenario, which ignored the information in multi-utterance turn and a more complex and natural scenario of multi-speaker conversations. This paper introduces a context information modeling approach that considers potential emotional interactive information within a speaker’s multi-utterance turn, which dominates his/her future emotions. Moreover, our approach advances emotion prediction in both dual-and multi-speaker conversations. Experimental results show that such an approach significantly enhances context information modeling and renders a higher accuracy in EPC than reported in the literature.

Emotion Awareness in Multi-utterance Turn for Improving Emotion Prediction in Multi-Speaker Conversation

Multimodal emotion recognition (MER) is a rapidly evolving field aimed at integrating information from various modalities, such as speech and text, to deepen our understanding of emotions. However, challenges in feature extraction and fusion hinder further advancements in MER performance. To address these challenges, we propose a MER method using self-supervised representations and handcrafted music theory-inspired representations across different modalities to comprehensively capture emotional information. Additionally, we introduce a novel multimodal fusion method to explore modality-specific and modality-invariant relationships, thereby reducing distribution gaps between different modalities in MER. Extensive experimental validation underscores the effectiveness of our approach, with state-of-the-art results showing a 3.55% improvement compared with the baseline. These results validate the effectiveness of our proposed method, signifying a notable enhancement in MER performance. 

Multimodal Fusion of Music Theory-Inspired and Self-Supervised Representations for Improved Emotion Recognition

This paper proposes an efficient attempt to noisy speech emotion recognition (NSER). Conventional NSER approaches have proven effective in mitigating the impact of artificial noise sources, such as white Gaussian noise, but are limited to non-stationary noises in real-world environments due to their complexity and uncertainty. To overcome this limitation, we introduce a new method for NSER by adopting the automatic speech recognition (ASR) model as a noise-robust feature extractor to eliminate non-vocal information in noisy speech. We first obtain intermediate layer information from the ASR model as a feature representation for emotional speech and then apply this representation for the downstream NSER task. Our experimental results show that 1) the proposed method achieves better NSER performance compared with the conventional noise reduction method, 2) outperforms self-supervised learning approaches, and 3) even outperforms text-based approaches using ASR transcription or the ground truth transcription of noisy speech.

pdf/on-the-effectiveness-of-asr-representations-in-real-world-3llvovdchx.pdf

On the Effectiveness of ASR Representations in Real-world Noisy Speech Emotion Recognition

Introduction. The article presents the results of an empirical study of subjective predictors of speech actions. The purpose of the article is to substantiate the indicators of students’ agency in learning on the basis of empirical research in subjective predictors of speech actions. Materials and Methods. The study follows A. N. Leontiev’s activity-based approach, A. A. Verbitsky’s theory of context education, and J. Kelly’s cognitive theory of personality. The research methods include the following: analysis, synthesis, generalization, classification, J. Kelly’s method of repertoire gratings, methods of mathematical statistics. Moreover, the author used co-therapeutic computer system «KELLI-98». Results. The empirical study on the unity of (connection between) subjective predictors of speech actions, which include significant roles, conjugations of the I-real/I-ideal, and the type of communication allowed to reveal four kinds of students’ agency in learning. The first one is a normal type of communication, characterized by manifested agency in interpersonal interaction with all subjects of the educational process, and speech action at the dialogue level. The second is a normal type of communication characterized by latent agency in interpersonal interaction with all subjects of the educational process; speech action at this level is an internal dialogue. The third type is a rigid type of communication distinguished by lack of objectivity in interpersonal interaction with all subjects of the educational process; speech action is characterized by opposition to others and resistance. The fourth type of communication is a rigid one. Its distinctive features include a poorly developed ability to solve problems in interpersonal interaction with all subjects of the educational process; speech actions include manipulation. Conclusions. Finally, relying on the empirical data analysis, the author concludes that the unity of subjective predictors of speech actions (significant roles, conjugation of the I-real/I-ideal, type of communication) is an indicator of ‘activity/inactivity’; on the basis of this indicator, four kinds of students’ learning behaviors are described.

The unity of subjective predictors of speech actions as an indicator of the subject’s agency in learning: An empirical study

Emotion prediction in conversation is important for humans to conduct a fluent conversation, which is an underexplored research topic in the affective computing area. In previous studies, predicting the coming emotion only considered the context information from one single speaker. However, there are two sides of the speaker and listener in interlocutors, and their emotions are influenced by one another during the conversation. For this reason, we propose a dimensional emotion prediction model based on interactive information in conversation from both interlocutors. We investigate the effects of interactive information in four conversation situations on emotion prediction, in which emotional tendencies of interlocutors are consistent or inconsistent in both valence and arousal. The results showed that the proposed method performance better by considering the interactive context information than the ones considering one single side alone. The prediction result is affected by the conversation situations. In the situation interlocutors have consistent emotional tendency in valence and inconsistent tendency in arousal, the prediction performance of valence is the best. In the situation that interlocutors’ emotional tendency is inconsistent in both valence and arousal, the prediction performance of arousal is the best.

Dimensional Emotion Prediction Based on Interactive Context in Conversation.

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Sixia Li

Author Tools

Chat about Author

Papers

Dimensional Emotion Prediction Based on Interactive Context in Conversation.