SEMAINE database

A key goal for the next generation of Human Computer Interaction devices is “really natural language processing”, which allows human users to speak to machines just as they would speak to another person. Achieving that goal depends on finding ways to handle the non-verbal signals that are part and parcel of human conversation. These in- clude signals associated with conversation management (e.g. backchannelling and turn-taking) and the emotional colouring of everyday communication (outbursts of “pure” emotion are a different problem).

A major issue hindering new developments in this field is the lack of databases with displays of this type of natural conversational behaviour and affect. To address this issue, the SEMAINE consortium recorded a very large database of conversations between a human user and a human operator who pretends to be an artificial intelligent agent with a prototypic emotional character. The operators pretend to be either Poppy, who is always happy, Spike, who is angry, Obadiah, who is depressed and gloomy, or Prudence who is very sensible. The operators endeavour to pull the users into their emotional quadrant, thus colouring the emotional content of the conversations. The conversational software can be downloaded from the SEMAINE wiki.


At this moment, the database contains approximately 240 character conversations, and recording is still ongoing. Currently approximately 80 conversations have been fully annotated for a number of dimensions in a fully continuous way using FeelTrace. The same conversations have also been fully transcribed. The database is freely available to the research community from http://semaine-db.eu. For more information about the database, please read [1].

Involved group members

Maja Pantic, Michel Francois Valstar

Related Publications

  1. The SEMAINE database: Annotated multimodal records of emotionally coloured conversations between a person and a limited agent

    G. Mckeown, M. F. Valstar, R. Cowie, M. Pantic, M. Schroeder. IEEE Transactions on Affective Computing. 3: pp. 5 - 17, Issue 1. April 2012.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @article{mckeown2011semaine,
        author = {G. Mckeown and M. F. Valstar and R. Cowie and M. Pantic and M. Schroeder},
        pages = {5--17},
        journal = {IEEE Transactions on Affective Computing},
        month = {April},
        note = {Issue 1},
        title = {The SEMAINE database: Annotated multimodal records of emotionally coloured conversations between a person and a limited agent},
        volume = {3},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Journal Article
    %T The SEMAINE database: Annotated multimodal records of emotionally coloured conversations between a person and a limited agent
    %A Mckeown, G.
    %A Valstar, M. F.
    %A Cowie, R.
    %A Pantic, M.
    %A Schroeder, M.
    %J IEEE Transactions on Affective Computing
    %D 2012
    %8 April
    %V 3
    %F mckeown2011semaine
    %O Issue 1
    %P 5-17

  2. Come and Have an Emotional Workout with Sensitive Artificial Listeners!

    M. Schroder, S. Pammi, H. Gunes, M. Pantic, M. F. Valstar, R. Cowie, G. Mckeown, D. Heylen, M. Maat, F. Eyben, B. Schuller, M. Wollmer, E. Bevacqua, C. Pelachaud, E. Sevin. Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG'11). Santa Barbara, CA, USA, pp. 646, March 2011.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{SchroderEtAlFG2011,
        author = {M. Schroder and S. Pammi and H. Gunes and M. Pantic and M. F. Valstar and R. Cowie and G. Mckeown and D. Heylen and M. Maat and F. Eyben and B. Schuller and M. Wollmer and E. Bevacqua and C. Pelachaud and E. Sevin},
        pages = {646},
        address = {Santa Barbara, CA, USA},
        booktitle = {Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG'11)},
        month = {March},
        title = {Come and Have an Emotional Workout with Sensitive Artificial Listeners!},
        year = {2011},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Come and Have an Emotional Workout with Sensitive Artificial Listeners!
    %A Schroder, M.
    %A Pammi, S.
    %A Gunes, H.
    %A Pantic, M.
    %A Valstar, M. F.
    %A Cowie, R.
    %A Mckeown, G.
    %A Heylen, D.
    %A Maat, M.
    %A Eyben, F.
    %A Schuller, B.
    %A Wollmer, M.
    %A Bevacqua, E.
    %A Pelachaud, C.
    %A Sevin, E.
    %B Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG?11)
    %D 2011
    %8 March
    %C Santa Barbara, CA, USA
    %F SchroderEtAlFG2011
    %P 646

  3. The SEMAINE corpus of emotionally coloured character interactions

    G. Mckeown, M. F. Valstar, R. Cowie, M. Pantic. Proceedings of IEEE Int'l Conf. Multimedia, Expo (ICME'10), Singapore. pp. 1079 - 1084, July 2010.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{McKeown2010tscoe,
        author = {G. Mckeown and M. F. Valstar and R. Cowie and M. Pantic},
        pages = {1079--1084},
        booktitle = {Proceedings of IEEE Int'l Conf. Multimedia, Expo (ICME'10), Singapore},
        month = {July},
        title = {The SEMAINE corpus of emotionally coloured character interactions},
        year = {2010},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T The SEMAINE corpus of emotionally coloured character interactions
    %A Mckeown, G.
    %A Valstar, M. F.
    %A Cowie, R.
    %A Pantic, M.
    %B Proceedings of IEEE Int?l Conf. Multimedia, Expo (ICME?10), Singapore
    %D 2010
    %8 July
    %F McKeown2010tscoe
    %P 1079-1084

  4. A Demonstration of Audiovisual Sensitive Artificial Listeners

    M. Schroeder, E. Bevacqua, F. Eyben, H. Gunes, D. Heylen, M. Maat, S. Pammi, M. Pantic, C. Pelachaud, B. Schuller, E. Sevin, M. F. Valstar, M. Woellmer. Proc. of ACII 2009, Proceedings of IEEE Int'l Conf. Affective Computing and Intelligent Interfaces (ACII'09). Amsterdam, Netherlands, 1: pp. 263 - 264, September 2009.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{Schroeder2009adoas,
        author = {M. Schroeder and E. Bevacqua and F. Eyben and H. Gunes and D. Heylen and M. Maat and S. Pammi and M. Pantic and C. Pelachaud and B. Schuller and E. Sevin and M. F. Valstar and M. Woellmer},
        pages = {263--264},
        address = {Amsterdam, Netherlands},
        booktitle = {Proc. of ACII 2009, Proceedings of IEEE Int'l Conf. Affective Computing and Intelligent Interfaces (ACII'09)},
        month = {September},
        title = {A Demonstration of Audiovisual Sensitive Artificial Listeners},
        volume = {1},
        year = {2009},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T A Demonstration of Audiovisual Sensitive Artificial Listeners
    %A Schroeder, M.
    %A Bevacqua, E.
    %A Eyben, F.
    %A Gunes, H.
    %A Heylen, D.
    %A Maat, M.
    %A Pammi, S.
    %A Pantic, M.
    %A Pelachaud, C.
    %A Schuller, B.
    %A Sevin, E.
    %A Valstar, M. F.
    %A Woellmer, M.
    %B Proc. of ACII 2009, Proceedings of IEEE Int?l Conf. Affective Computing and Intelligent Interfaces (ACII?09)
    %D 2009
    %8 September
    %V 1
    %C Amsterdam, Netherlands
    %F Schroeder2009adoas
    %P 263-264