We developed a system, named Gaze-X, to support affective multimodal human-computer interaction (AMM-HCI) where the user’s actions and emotions are modeled and then used to adapt the interaction and support the user in his or her activity.
The system is based on sensing and interpretation of the human part of the computer’s context defined as W5+ (who, where, what, when, why, how). It integrates a number of natural human communicative modalities including speech, eye gaze direction, face and facial expression, and a number of standard HCI modalities like keystrokes, mouse movements, and active software identification, which, in turn, are fed into processes that provide decision making and adapt the HCI to support the user in his or her activity according to his or her preferences. A usability study conducted in an office scenario with a number of users indicates that Gaze-X is perceived as effective, easy to use, useful, and affectively qualitative.
J. Shen, W. Shi, M. Pantic. Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG'11), ECAG'11- 2nd FG Workshop on Facial and Bodily Expressions for Control and Adaptation of Games . Santa Barbara, CA, USA, pp. 766 - 773, March 2011.
M. Pantic, A. Nijholt, A. Pentland, T. Huang. Int'l Journal of Autonomous and Adaptive Communications Systems. 1(2): pp. 168 - 187, 2008.
M. Santos, M. Pantic. Proceedings of Int'l Conf. Interfaces and Human Computer Interaction (IHCI'08). Amsterdam, Netherlands, pp. 101 - 108, July 2008.
L. Maat, M. Pantic. editors: T. Huang, A. Nijholt, A. Pentland, M. Pantic. Lecture Notes in Artificial Intelligence, Special Volume on Artificial Intelligence for Human Computing. vol. 4451, pp. 251 - 271, 2007.
L. Maat, M. Pantic. Proceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'06). Banff, Canada, pp. 171 - 178, November 2006.
M. Pantic. Key Note Paper, Lecture Notes in Artificial Intelligence, Special Volume on Ambient Intelligence in Everyday Life. vol. 3864, pp. 35 - 66, 2006.
M. Pantic, L. Rothkrantz. Invited Paper, Proceedings of the IEEE, Special Issue on Multimodal Human-Computer Interaction (HCI). 91(9): pp. 1370 - 1390, 2003.
M. Pantic, L. Rothkrantz. Proceedings of AAAI / IEEE Int'l Conf. Enterprise Information Systems (ICEIS'01). Setubal, Portugal, pp. 466 - 474, July 2001.