Social signal processing

Social signal processing, the detection and interpretation of temporal patterns of non—verbal behavioral cues in a given context, is a natural and, for the most part, unconscious process for humans. This process remains a rather difficult task for machines, but it is an important one to achieve if the goal is to realize a naturalistic and efficient human-computer interaction.

Agreements and disagreements are social attitudes that occur daily and are inevitable in a variety of everyday situations, from everyday simple discussions over, e.g., which restaurant to dine in, to traditionally controversial topics like politics. As a first step towards realizing the goal of a socially sensitive machine, we have focused our efforts towards devising a way to automatically detect these social attitudes of agreement and disagreement.

This work is our first attempt towards the automatic detection of spontaneous agreement and disagreement [1]. Related work has used only verbal and prosodic features. This work-in-progress attempts to show that:

  1. Nonverbal behavior alone is enough to distinguish between pre-segmented episodes of agreement and disagreement, 
  2. incorporating gestural information improves the accuracy of classifying such episodes.

 


The data used are video segments annotated as agreement or disagreement, from the Canal9 Political Debates, a database of real televised debates, containing highly spontaneous data. Only episodes of direct (dis)agreement are used, i.e. segments during which the speaker specifically states her agreement or disagreement. The cues used are the relevant head and hand gestures, as those are listed in our survey on (dis)agreement, which have been manually annotated in all the episodes of agreement and disagreement we use, and the prosodic features used in related work (fundamental frequency and energy).

Involved group members

Maja Pantic, Konstantinos Bousmalis, Aaron C. Elkins, Ognjen Rudovic

Related Publications

  1. Bridging the gap between social animal and unsocial machine: A survey of Social Signal Processing

    A. Vinciarelli, M. Pantic, D. Heylen, C. Pelachaud, I. Poggi, F. D'ericco, M. Schroeder. IEEE Transactions on Affective Computing. 3: pp. 69 - 87, Issue 1. April 2012.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @article{vinciarelli2011bridging,
        author = {A. Vinciarelli and M. Pantic and D. Heylen and C. Pelachaud and I. Poggi and F. D'ericco and M. Schroeder},
        pages = {69--87},
        journal = {IEEE Transactions on Affective Computing},
        month = {April},
        note = {Issue 1},
        title = {Bridging the gap between social animal and unsocial machine: A survey of Social Signal Processing},
        volume = {3},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Journal Article
    %T Bridging the gap between social animal and unsocial machine: A survey of Social Signal Processing
    %A Vinciarelli, A.
    %A Pantic, M.
    %A Heylen, D.
    %A Pelachaud, C.
    %A Poggi, I.
    %A D?ericco, F.
    %A Schroeder, M.
    %J IEEE Transactions on Affective Computing
    %D 2012
    %8 April
    %V 3
    %F vinciarelli2011bridging
    %O Issue 1
    %P 69-87

  2. Social Signal Processing: The Research Agenda

    M. Pantic, R. Cowie, F. D'ericco, D. Heylen, M. Mehu, C. Pelachaud, I. Poggi, M. Schroder, A. Vinciarelli. Visual Analysis of Humans. pp. 511 - 538, 2011.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inbook{PanticEtAlVAOH2011,
        author = {M. Pantic and R. Cowie and F. D'ericco and D. Heylen and M. Mehu and C. Pelachaud and I. Poggi and M. Schroder and A. Vinciarelli},
        pages = {511--538},
        booktitle = {Visual Analysis of Humans},
        title = {Social Signal Processing: The Research Agenda},
        year = {2011},
    }
    Endnote reference [hide]
    %0 Book Section
    %A Pantic, M.
    %A Cowie, R.
    %A D?ericco, F.
    %A Heylen, D.
    %A Mehu, M.
    %A Pelachaud, C.
    %A Poggi, I.
    %A Schroder, M.
    %A Vinciarelli, A.
    %B Visual Analysis of Humans: The Research Agenda
    %D 2011
    %F PanticEtAlVAOH2011
    %P 511-538

  3. Modeling Hidden Dynamics of Multimodal Cues for Spontaneous Agreement and Disagreement Recognition

    K. Bousmalis, L. Morency, M. Pantic. Proc. IEEE Int'l Conf. on Automatic Face and Gesture Recognition. pp. 746 - 752, Oral. March 2011.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{FG11Bousmalis,
        author = {K. Bousmalis and L. Morency and M. Pantic},
        pages = {746--752},
        booktitle = {Proc. IEEE Int'l Conf. on Automatic Face and Gesture Recognition},
        month = {March},
        note = {Oral},
        title = {Modeling Hidden Dynamics of Multimodal Cues for Spontaneous Agreement and Disagreement Recognition},
        year = {2011},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Modeling Hidden Dynamics of Multimodal Cues for Spontaneous Agreement and Disagreement Recognition
    %A Bousmalis, K.
    %A Morency, L.
    %A Pantic, M.
    %B Proc. IEEE Int?l Conf. on Automatic Face and Gesture Recognition
    %D 2011
    %8 March
    %F FG11Bousmalis
    %O Oral
    %P 746-752

  4. Social Signal Processing: Survey of an Emerging Domain

    A. Vinciarelli, M. Pantic, H. Bourlard. Image and Vision Computing Journal. 27(12): pp. 1743 - 1759, 2009.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @article{Vinciarelli2009sspso,
        author = {A. Vinciarelli and M. Pantic and H. Bourlard},
        pages = {1743--1759},
        journal = {Image and Vision Computing Journal},
        number = {12},
        title = {Social Signal Processing: Survey of an Emerging Domain},
        volume = {27},
        year = {2009},
    }
    Endnote reference [hide]
    %0 Journal Article
    %T Social Signal Processing: Survey of an Emerging Domain
    %A Vinciarelli, A.
    %A Pantic, M.
    %A Bourlard, H.
    %J Image and Vision Computing Journal
    %D 2009
    %V 27
    %N 12
    %F Vinciarelli2009sspso
    %P 1743-1759

  5. Spotting Agreement and Disagreement: A Survey of Nonverbal Audiovisual Cues and Tools

    K. Bousmalis, M. Mehu, M. Pantic. Proceedings of IEEE Int'l Conf. Affective Computing and Intelligent Interfaces (ACII'09). Amsterdam, Netherlands, 2: pp. 1 - 9, September 2009.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{Bousmalis2009saada,
        author = {K. Bousmalis and M. Mehu and M. Pantic},
        pages = {1--9},
        address = {Amsterdam, Netherlands},
        booktitle = {Proceedings of IEEE Int'l Conf. Affective Computing and Intelligent Interfaces (ACII'09)},
        month = {September},
        title = {Spotting Agreement and Disagreement: A Survey of Nonverbal Audiovisual Cues and Tools},
        volume = {2},
        year = {2009},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Spotting Agreement and Disagreement: A Survey of Nonverbal Audiovisual Cues and Tools
    %A Bousmalis, K.
    %A Mehu, M.
    %A Pantic, M.
    %B Proceedings of IEEE Int?l Conf. Affective Computing and Intelligent Interfaces (ACII?09)
    %D 2009
    %8 September
    %V 2
    %C Amsterdam, Netherlands
    %F Bousmalis2009saada
    %P 1-9

  6. Social Signal Processing: Understanding Social Actions through nonverbal behaviour analysis

    A. Vinciarelli, H. Salamin, M. Pantic. Proceedings of IEEE Int'l Conf. Computer Vision and Pattern Recognition (CVPR-W'09). Miami, Florida, USA, 3: pp. 42 - 49, June 2009.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{Vinciarelli2009sspus,
        author = {A. Vinciarelli and H. Salamin and M. Pantic},
        pages = {42--49},
        address = {Miami, Florida, USA},
        booktitle = {Proceedings of IEEE Int'l Conf. Computer Vision and Pattern Recognition (CVPR-W'09)},
        month = {June},
        title = {Social Signal Processing: Understanding Social Actions through nonverbal behaviour analysis},
        volume = {3},
        year = {2009},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Social Signal Processing: Understanding Social Actions through nonverbal behaviour analysis
    %A Vinciarelli, A.
    %A Salamin, H.
    %A Pantic, M.
    %B Proceedings of IEEE Int?l Conf. Computer Vision and Pattern Recognition (CVPR-W?09)
    %D 2009
    %8 June
    %V 3
    %C Miami, Florida, USA
    %F Vinciarelli2009sspus
    %P 42-49

  7. Social Signals, Their Function and Automatic Analysis: A Survey

    A. Vinciarelli, M. Pantic, H. Bourlard, A. Pentland. Proceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'08). Chania, Greece, pp. 61 - 68, October 2008.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{Vinciarelli2008sstfa,
        author = {A. Vinciarelli and M. Pantic and H. Bourlard and A. Pentland},
        pages = {61--68},
        address = {Chania, Greece},
        booktitle = {Proceedings of ACM Int'l Conf. Multimodal Interfaces (ICMI'08)},
        month = {October},
        title = {Social Signals, Their Function and Automatic Analysis: A Survey},
        year = {2008},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Social Signals, Their Function and Automatic Analysis: A Survey
    %A Vinciarelli, A.
    %A Pantic, M.
    %A Bourlard, H.
    %A Pentland, A.
    %B Proceedings of ACM Int?l Conf. Multimodal Interfaces (ICMI?08)
    %D 2008
    %8 October
    %C Chania, Greece
    %F Vinciarelli2008sstfa
    %P 61-68

  8. Social Signal Processing: State of the Art and Future Perspectives of an Emerging Domain

    A. Vinciarelli, M. Pantic, H. Bourlard, A. Pentland. Proceedings of ACM Int'l Conf. Multimedia (MM'08). Vancouver, Canada, pp. 1061 - 1070, October 2008.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{Vinciarelli2008sspso,
        author = {A. Vinciarelli and M. Pantic and H. Bourlard and A. Pentland},
        pages = {1061--1070},
        address = {Vancouver, Canada},
        booktitle = {Proceedings of ACM Int'l Conf. Multimedia (MM'08)},
        month = {October},
        title = {Social Signal Processing: State of the Art and Future Perspectives of an Emerging Domain},
        year = {2008},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Social Signal Processing: State of the Art and Future Perspectives of an Emerging Domain
    %A Vinciarelli, A.
    %A Pantic, M.
    %A Bourlard, H.
    %A Pentland, A.
    %B Proceedings of ACM Int?l Conf. Multimedia (MM?08)
    %D 2008
    %8 October
    %C Vancouver, Canada
    %F Vinciarelli2008sspso
    %P 1061-1070