Mainstream automatic speech recognition has focused almost exclusively on the acoustic signal. The performance of these systems degrades considerably in the real word in the presence of noise. It was needed novel approaches that use other orthogonal sources of information to the acoustic input that not only considerably improve the performance in severely degraded conditions, but also are independent to the type of noise and reverberation. Visual speech is one such source not perturbed by the acoustic environment and noise. In this paper, it was presented own approach to lip-tracking and fusion of signals audio and video for audio-visual speech and speaker recognition system. It was presented video analysis of visual speech for extraction visual features from a talking person in color video sequences. It was developed a method for automatically localization of face, eyes, region of mouth, corners and contour of mouth. It was proposed synchronous and two asynchronous of methods of fusion of signals audio and video. Finally, the paper will show results of lip-tracking depending on various factors (lighting, beard), results of speech and speaker recognition in noisy environments.

Method of Speech Recognition and Speaker Identification using Audio-Visual of Polish Speech and Hidden Markov Models

The use of soft computing for virtual object implementation is a novel technique that has received a lot of attention in the last few years. This field is very useful especially for applications related to computer games and virtual environments. One of the application areas is the use of real time lip tracking information for the implementation of the virtual lips in animated characters. The increased computing power of computers has made it possible to do real time lip tracking. This real time lip tracking information can be used to implement and control a virtual lip. On the other hand, the use of soft computing to represent the real time lip parameters enables us to have a more robust and flexible system which can compensate for the potential errors of lip tracking. In this paper we present a system which generates real time lip information for soft computing based virtual animated lips.

Real-time lip tracking for virtual lip implementation in virtual environments and computer games

The paper introduces an annotation scheme for a political debate dataset which is mainly in the form of video, and audio annotations. The annotation contains various infor- mation ranging from general linguistic to domain specific information. Some are annotated with automatic tools, and some are manually annotated. One of the goals is to use the information to predict the categories of the answers by the speaker to the disruptions. A typology of such answers is proposed and an automatic categorization system based on a multimodal parametrization is successfully performed.

https://hal.archives-ouvertes.fr/hal-00983742/document

Multimodal Annotations and Categorization for Political Debates

La reconnaissance des emotions est un aspect important de l'informatique affective dont l'un des objectifs est l'etude et le developpement d'interactions comportementales et emotionnelles entre humains et agents conversationnels animes. Dans ce contexte, un point egalement important concerne les dispositifs d'acquisition et les outils de traitement des signaux, conduisant a une estimation de l'etat emotionnel de l'utilisateur. Le travail presente dans ce manuscrit repose sur le developpement d'une plate-forme multimodale d'acquisition et de traitement de signaux physiologiques (PACE). Cette plate-forme peut etre consideree comme un middleware modulaire et generique permettant de mesurer, en temps reel, l'etat emotionnel de l'utilisateur via l'estimation d'un vecteur d'etat. Elle s'integre dans le cadre de la mise en place d'un simulateur innovant destine a la simulation d'entretiens d'embauche (Projet PISE : Plate-forme Immersive de Simulation d'Entretien). Cet outil permettra aux personnes en phase d'insertion ou de reinsertion, une meilleure gestion de leurs competences comportementales et emotionnelles. L'approche que nous avons developpee se fonde sur les resultats experimentaux obtenus et sur une methodologie originale impliquant differents modeles mathematiques. Differents protocoles experimentaux, bases sur deux techniques d'induction de stimuli (images IAPS et sequences video de Schaeffer) ont ete proposes. Ils permettent la determination de la correlation entre les etats emotionnels et les signaux physiologiques (EEG, ECG, etc.) issus des capteurs proposes. Trois methodes de classification (SVM, naive bayesienne, regression logistique) ont ete comparees sur plusieurs criteres d'evaluation. Nous avons propose, a partir des resultats obtenus, une methodologie permettant l'extraction en temps reel des emotions via les modeles proposes. Une etude experimentale a egalement ete menee avec pour objectif de valider la plate-forme PACE via la reconnaissance d'etats emotionnels lors de sequences video developpees a l'Universite d'Angers. L'approche multimodale proposee a donne de meilleurs resultats que les approches uni-modales precedentes. Enfin, notre plate-forme a ete integree au simulateur PISE, et evaluee de maniere subjective et objective lors de simulations d'entretiens. Les resultats ont permis de valider partiellement le simulateur.

https://tel.archives-ouvertes.fr/tel-00997249/document

Plate-forme multimodale pour la reconnaissance d'émotions via l'analyse de signaux physiologiques : Application à la simulation d'entretiens d'embauche

In this paper, we present an Audio-Visual Automatic Speech Recognition System that combines the acoustic and the visual data. The proposed algorithm here, for modelling the multimodal data, is a Hidden Markov Model (HMM) hybridised with the Genetic Algorithm (GA) to determine its optimal structure. This algorithm is combined with the Baum-Welch algorithm, which allows an effective re-estimation of the probabilities of the HMM. Our experiments show the improvement in the performance of the most promising audio-visual system, based on the combination of GA/HMM model compared to the traditional HMM.

Evolutionary structure of hidden Markov models for audio-visual Arabic speech recognition

Bimodal speech recognition is a robust technique for automated speech analysis, and has received a lot of attention in the last few decades. In this paper, we analyze the effect of the HMM models on the performance of the bimodal speech recognizer, present a comparative analysis of the different HMM models that can be used in bimodal speech recognition, and finally propose a novel model, which has been experimentally verified to perform better than others. One of the unique characteristic of our HMM model is the novel fusion strategy of the acoustic and the visual features, that takes into account the different sampling rates of these two signals. Compared to audio only, the bimodal speech recognition scheme has a much more improved recognition accuracy, especially in presence of noise.

HMM modeling for audio-visual speech recognition

Popular Tasks used by Researchers

How can I help with your research?

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

We encountered an error while loading the panel. Please try refreshing the page.

Something went wrong

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

Anyone with the link can view this chat. You can switch it back to private anytime.

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

C.C. Ko

Author Tools

Chat about Author

Papers

HMM modeling for audio-visual speech recognition