Multimodal Interfaces: A Survey of Principles, Models and Frameworks

Question

1. What have the authors contributed in "Multimodal interfaces: a survey of principles, models and frameworks" ?

2. What have the authors stated for future works in "Multimodal interfaces: a survey of principles, models and frameworks" ?

3. What are the main advantages of multimodal interfaces?

4. What is the main objective of IM2?

Accepted Answer

This opens a number of associated issues covered by this chapter, such as heterogeneous data types fusion, architectures for real-time processing, dialog management, machine learning for multimodal interaction, modeling languages, frameworks, etc.. The article concludes with an outline of the current state of multimodal interaction research in Switzerland, and also summarizes the major future challenges in the field.

Accepted Answer

Although many issues have been addressed well in the multimodal interaction research and systems literature, such as fusion of heterogeneous data types, architectures for real-time processing, dialog management, map-based multimodal interaction, and so forth, nonetheless the field is still young and needs further research to build reliable multimodal systems and usable applications.. But future work clearly is needed to work toward the design of usable adaptive multimodal interfaces.. Multimodal dialog processing also will gain in the future from the recent and promising subfield of social signal processing, which can assist dialog modeling by providing a dialog manager with real-time information about a given user ’ s state and her current social and collaborative context.. Other important future directions for multimodal research include human/machine interaction using new tangible interfaces such as digital paper and pen, and multitouch tables, surfaces and screens.

Accepted Answer

Multimodal systems have the potential to enhance human/computer interaction in a number of ways: • Enhanced robustness due to combining different partial information sources; • Flexible personalization based on user and context; • New functionality involving multi-user and mobile interaction.

Accepted Answer

IM2 aims at developing natural multimodal interfaces for human-computer interaction and to foster collaboration, focusing on new multimodal technologies to support human interaction, in the context of smart meeting rooms and remote meeting assistants.

Accepted Answer

Other important future directions for multimodal research include human/machine interaction using new tangible interfaces such as digital paper and pen, and multitouch tables, surfaces and screens.

Accepted Answer

Specific guidelines include integrating complementary modalities to improve system robustness, and giving users better control over modality selection so they can avoid errors.

Accepted Answer

A last point that stems from these six guidelines is readability: a language for description of multimodal interaction should be readable, as much in regard to the machine as to humans.

Accepted Answer

Due to the multiple recognizers necessary to interpret multimodal input and the continuous property of input streams, multimodal systems depend on time synchronized parallel processing.

Accepted Answer

HephaisTK can be configured with the SMUIML language (Synchronized Multimodal User Interfaces Markup Language) [19], allowing a clear description of the human-machine multimodal dialog and control over the way multiple input modalities have to be fused.

Accepted Answer

The HephaisTK toolkit has been designed to plug itself in a client application that wishes to receive notifications of multimodal events received from a set of modality recognizers.

Accepted Answer

as the authors will see in the following section, the time sensitivity of multimodal systems is crucial to determining the order of processing multimodal commands in parallel or in sequence.

Accepted Answer

The time constraint is highly important in multimodal systems and all the modalities should be properly time-stamped and synchronized.

Accepted Answer

Multimodal systems can be very resource demanding in some cases (e.g., speech/gesture recognition, machine-learning augmented integration).

Accepted Answer

Adaptability can then be addressed with the help of machine learning, by watching the users’ behavior in the sensed context [21].

Accepted Answer

On one hand, multimodal interfaces target a more “human” way of interacting with computers, by means of speech, gestures or other modalities, as well as being preferred over unimodal interfaces by users [49]; on the other hand, multimodal interfaces have been demonstrated to offer better flexibility and reliability than other human/machine interaction means [51].

Multimodal Interfaces: A Survey of Principles, Models and Frameworks

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What have the authors contributed in "Multimodal interfaces: a survey of principles, models and frameworks" ?

2. What have the authors stated for future works in "Multimodal interfaces: a survey of principles, models and frameworks" ?

3. What are the main advantages of multimodal interfaces?

4. What is the main objective of IM2?

5. What are the main future directions for multimodal research?

6. What are the specific guidelines for integrating complementary modalities?

7. What is the last point that stems from these six guidelines?

8. Why are multimodal interfaces considered to be more efficient?

9. What language can be used to configure a multimodal interface?

10. What is the purpose of the HephaisTK toolkit?

11. What is the importance of multimodal systems?

12. What is the importance of time constraint in multimodal systems?

13. What are some of the advantages of multimodal systems?

14. What is the way to address Adaptability?

15. What is the difference between multimodal and unimodal interfaces?

Figures

Citations

Multimodal Software For Affective Education: User Interaction Design And Evaluation

M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

Improving Interaction with TV-Based Applications through Adaptive Multimodal Fission

Ludic Educational Game Creation Tool: Teaching Schoolers Road Safety

Multimodal User Interface in E-Learning Environments: A Proposed Architecture

References

Chapter 11 Working memory

The Design of Everyday Things

Design of Everyday Things

The Psychology of Human-Computer Interaction

Hand and Mind: What Gestures Reveal about Thought

Related Papers (5)

“Put-that-there”: Voice and gesture at the graphics interface

Multimodal human-computer interaction: A survey

Advances in robust multimodal interface design

Ten myths of multimodal interaction

Review Article: Multimodal interaction: A review