Emotion recognition is an aspect of human-computer interaction, affective computing, and social robotics. Conventional unimodal approaches for emotion recognition, depending on single data sources such as facial expressions or speech signals often fall short in capturing the complexity and context-dependent nature of emotions. Multimodal Emotion Recognition (MER), which integrates information from multiple modalities, has emerged as a promising solution to overcome these limitations. In recent years, Transformers-based approaches have gathered significant attention in the fields of natural language processing and computer vision, highlighting their ability to capture long-range dependencies and semantic representations. These models have rapidly achieved the MER state-of-the-art. However, current survey papers that cover MER lack a specific focus on Transformer-based techniques. To bridge this research gap, this review paper provides a comprehensive investigation of Transformers-based approaches for MER. It explores various Transformer architectures and proposes several scenarios for using Transformers at different stages of MER process. In addition, it examines datasets suitable for MER, discusses fusion mechanisms, and introduces novel taxonomies in both MER and Transformer technologies. The review also addresses challenges and future research directions. Through this review, we aim to provide researchers with an inclusive understanding of the current state-of-the-art in Transformers-based approaches for MER, paving the way for further advancements in this rapidly developing field. 

Using transformers for multimodal emotion recognition: Taxonomies and state of the art review

Wearable devices or wearables, designed to be attached to the human body, can gather personalized real-time data and continuously monitor an individual’s health status and physiological disposition in a non-invasive manner. Intelligent wearables integrate advanced machine learning (ML) algorithms to process complex data patterns and provide accurate insights. As a result, intelligent wearables have emerged as a groundbreaking innovation in the fields of sports and health, introducing a new paradigm in kinematic analysis and patient data evaluation. For example, virtual coaches offer feedback on athletes’ performance, while virtual physicians assist in customizing medication for patients. This article provides an overview of various types of intelligent wearables and their applications in health and sports, categorizes ML algorithms, and introduces the wireless body area sensor network (WBASN) used for communication in wearable sensors. Additionally, we discuss potential challenges and development directions that could shape the future of intelligent wearables and propose effective solutions for their continued enhancement. This article offers valuable insights into the exciting potential of intelligent wearables to transform healthcare and sports.

Intelligent Wearable Systems: Opportunities and Challenges in Health and Sports

STSNet: a novel spatio-temporal-spectral network for subject-independent EEG-based emotion recognition

The dynamic expressions of emotion convey both the emotional and functional states of an individual’s interactions. Recognizing the emotional states helps us understand human feelings and thoughts. Systems and frameworks designed to recognize human emotional states automatically can use various affective signals as inputs, such as visual, vocal and physiological signals. However, emotion recognition via a single modality can be affected by various sources of noise that are specific to that modality and the fact that different emotion states may be indistinguishable. This review examines the current state of multimodal emotion recognition methods that integrate visual, vocal or physiological modalities for practical emotion computing. Recent empirical evidence on deep learning methods used for fine-grained recognition is reviewed, with discussions on the robustness issues of such methods. This review elaborates on the profound learning challenges and solutions required for a high-quality emotion recognition system, emphasizing the benefits of dynamic expression analysis, which aids in detecting subtle micro-expressions, and the importance of multimodal fusion for improving emotion recognition accuracy. The literature was comprehensively searched via databases with records covering the topic of affective computing, followed by rigorous screening and selection of relevant studies. The results show that the effectiveness of current multimodal emotion recognition methods is affected by the limited availability of training data, insufficient context awareness, and challenges posed by real-world cases of noisy or missing modalities. The findings suggest that improving emotion recognition requires better representation of input data, refined feature extraction, and optimized aggregation of modalities within a multimodal framework, along with incorporating state-of-the-art methods for recognizing dynamic expressions. 

Multimodal Emotion Recognition Using Visual, Vocal and Physiological Signals: A Review

Human emotion recognition, crucial for interpersonal relations and human-building interaction, identifies emotions from various behavioral signals to improve user interactions. To enhance the performance of emotion recognition, this study proposed a novel model that fuses physiological, environmental, and personal data. A unique dataset was created via experiments conducted in an environmental chamber, and an emotion recognition model was subsequently developed using a multimodal fusion approach. The model transforms physiological data into 2D images to capture time series and spatial features and uniquely incorporates metadata, including environmental and personal data. The model's generalizability was validated using a leave-one-sample-out approach. The result showed 31.6% reduction of error with a predicted area when physiological, environmental, and personal data were fused in the emotion recognition model, suggesting that incorporating various contextual factors beyond physiological changes, such as the surrounding environment and inherent or acquired individual traits, can significantly enhance the model's understanding of emotions. Furthermore, the model was to be robust to individual differences, offering consistent emotion recognition across different subjects. These findings suggest that the proposed model can serve as a potent tool for emotion recognition in built environmental applications. 

Enhancing emotion recognition using multimodal fusion of physiological, environmental, personal data

Recently, wearable emotion recognition based on peripheral physiological signals has drawn massive attention due to its less invasive nature and its applicability in real-life scenarios. However, how to effectively fuse multimodal data remains a challenging problem. Moreover, traditional fully-supervised based approaches suffer from overfitting given limited labeled data. To address the above issues, we propose a novel self-supervised learning (SSL) framework for wearable emotion recognition, where efficient multimodal fusion is realized with temporal convolution-based modality-specific encoders and a transformer-based shared encoder, capturing both intra-modal and inter-modal correlations. Extensive unlabeled data is automatically assigned labels by five signal transforms, and the proposed SSL model is pre-trained with signal transformation recognition as a pretext task, allowing the extraction of generalized multimodal representations for emotion-related downstream tasks. For evaluation, the proposed SSL model was first pre-trained on a large-scale self-collected physiological dataset and the resulting encoder was subsequently frozen or fine-tuned on three public supervised emotion recognition datasets. Ultimately, our SSL-based method achieved state-of-the-art results in various emotion classification tasks. Meanwhile, the proposed model proved to be more accurate and robust compared to fully-supervised methods on low data regimes.

pdf/transformer-based-self-supervised-multimodal-representation-3dpe2kww.pdf

Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition

Emotion recognition plays an important role in human computer interaction systems as it helps the computer in understanding human behavior and their decision making process. Using Electroencephalographic (EEG) signals in emotion recognition offers a direct assessment on the inner state of human mind. This study aims to build a subject dependent emotion recognition system that differentiate between high and low levels of valance and arousal, using multidimensional EEG signals. Our system offers a transfer learning- minimum distance to Riemannian mean (TL-MDRM) framework. In this work, we perform two pre-processing stages. In the first stage, we analyze the EEG signals to investigate their non-Gaussianity and determine the most appropriate signal distribution. Using several statistical and goodness of fit tests, T-distribution was found to be the most appropriate distribution. Covariance matrix estimations plays a crucial step in manifold learning technique, based on the most suitable signal distribution the covariance matrix estimation technique is chosen. In the second stage, we perform transfer learning to deal with cross-session variability by generating a unique reference point for each participant and performing affine transformation for the covariance matrices on the symmetric positive definite (SPD) manifold around this point. The results show that, TL process improved the performance even when assuming Gaussian distribution, while assuming T-distribution with TL improved the performance further.

https://ieeexplore.ieee.org/ielx7/6287639/9668973/09704864.pdf

Subject-Dependent Emotion Recognition System Based on Multidimensional Electroencephalographic Signals: A Riemannian Geometry Approach

Existing multimodal stress/pain recognition approaches generally extract features from different modalities independently and thus ignore cross-modality correlations. This paper proposes a novel geometric framework for multimodal stress/pain detection utilizing Symmetric Positive Definite (SPD) matrices as a representation that incorporates the correlation relationship of physiological and behavioural signals from covariance and cross-covariance. Considering the non-linearity of the Riemannian manifold of SPD matrices, well-known machine learning techniques are not suited to classify these matrices. Therefore, a tangent space mapping method is adopted to map the derived SPD matrix sequences to the vector sequences in the tangent space where the LSTM-based network can be applied for classification. The proposed framework has been evaluated on two public multimodal datasets, achieving both the state-of-the-art results for stress and pain detection tasks.

Fusion of Physiological and Behavioural Signals on SPD Manifolds with Application to Stress and Pain Detection

Metric Learning on Complex Projective Spaces

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

Yujin Wu

Author Tools

Chat about Author

Papers

Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition

Subject-Dependent Emotion Recognition System Based on Multidimensional Electroencephalographic Signals: A Riemannian Geometry Approach

Fusion of Physiological and Behavioural Signals on SPD Manifolds with Application to Stress and Pain Detection

Metric Learning on Complex Projective Spaces