AU detection

The detection of facial muscle actions, as described in the Facial Action Coding System (FACS), is an ongoing effort with numerous potential applications. Examples of applications are online avatars mimicking a user's expression, automatic detection of user's feelings, and early detection of numerous diseases.

Our lab has a long history in detecting and analysing FACS Action Units (AUs). Early research aimed at detection of AU in static images based on facial component contour detection and an expert system encoding the FACS rules [1], [2], [3]. This was the first approach to address the problem of facial muscle action (AU) detection by combining the analysis of frontal and profile face view. Our subsequent research addressed AU detection in video sequences using Motion History Images [4], [5], and automatically tracked facial points in either profile [6], [7], or frontal facial view [8], [9]. This work was also the first to report on automatic recognition of temporal phases (onset, apex, offset) of facial actions in either frontal [8] or profile facial view [7]. Recent work addresses (near) real-time detection of AUs using appearance-based approaches relying on novel dynamic appearance descriptors such as LPQ-TOP [10]. 

Currently we are working on detection of AUs in natural conversational data, using novel dynamic appearance descriptors.

Related Publications

  1. Local Normal Binary Patterns for 3D Facial Action Unit Detection

    G. Sandbach, S. Zafeiriou, M. Pantic. Proceedings of the IEEE International Conference on Image Processing (ICIP 2012). Orlando, FL, USA, pp. 1813 - 1816, October 2012.

    Bibtex reference [hide]
    @inproceedings{sandbach2012normal,
        author = {G. Sandbach and S. Zafeiriou and M. Pantic},
        pages = {1813--1816},
        address = {Orlando, FL, USA},
        booktitle = {Proceedings of the IEEE International Conference on Image Processing (ICIP 2012)},
        month = {October},
        title = {Local Normal Binary Patterns for 3D Facial Action Unit Detection},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Local Normal Binary Patterns for 3D Facial Action Unit Detection
    %A Sandbach, G.
    %A Zafeiriou, S.
    %A Pantic, M.
    %B Proceedings of the IEEE International Conference on Image Processing (ICIP 2012)
    %D 2012
    %8 October
    %C Orlando, FL, USA
    %F sandbach2012normal
    %P 1813-1816

  2. Kernel Conditional Ordinal Random Fields for Temporal Segmentation of Facial Action Units

    O. Rudovic, V. Pavlovic, M. Pantic. Workshop on “What is in the Face”, Proceedings of the 12th European Conference on Computer Vision (ECCV-W'12), Part II. Florence, Italy, pp. 260 - 269, October 2012.

    Bibtex reference [hide]
    @inproceedings{RudovicEtAlECCV-W12,
        author = {O. Rudovic and V. Pavlovic and M. Pantic},
        pages = {260--269},
        address = {Florence, Italy},
        booktitle = {Workshop on “What is in the Face”, Proceedings of the 12th European Conference on Computer Vision (ECCV-W'12), Part II},
        month = {October},
        title = {Kernel Conditional Ordinal Random Fields for Temporal Segmentation of Facial Action Units},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Kernel Conditional Ordinal Random Fields for Temporal Segmentation of Facial Action Units
    %A Rudovic, O.
    %A Pavlovic, V.
    %A Pantic, M.
    %B Workshop on “What is in the Face”, Proceedings of the 12th European Conference on Computer Vision (ECCV-W?12), Part II
    %D 2012
    %8 October
    %C Florence, Italy
    %F RudovicEtAlECCV-W12
    %P 260-269

  3. Facial Action Detection using Block-based Pyramid Appearance Descriptors

    B. Jiang, M. F. Valstar, M. Pantic. Proceedings of the ASE/IEEE International Conference on Social Computing (SocialCom 2012). Amsterdam, The Netherlands, September 2012.

    Bibtex reference [hide]
    @inproceedings{jiang2012facial,
        author = {B. Jiang and M. F. Valstar and M. Pantic},
        address = {Amsterdam, The Netherlands},
        booktitle = {Proceedings of the ASE/IEEE International Conference on Social Computing (SocialCom 2012)},
        month = {September},
        title = {Facial Action Detection using Block-based Pyramid Appearance Descriptors},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Facial Action Detection using Block-based Pyramid Appearance Descriptors
    %A Jiang, B.
    %A Valstar, M. F.
    %A Pantic, M.
    %B Proceedings of the ASE/IEEE International Conference on Social Computing (SocialCom 2012)
    %D 2012
    %8 September
    %C Amsterdam, The Netherlands
    %F jiang2012facial

  4. Binary Pattern Analysis for 3D Facial Action Unit Detection

    G. Sandbach, S. Zafeiriou, M. Pantic. Proceedings of the British Machine Vision Conference (BMVC 2012). Guildford, UK, September 2012.

    Bibtex reference [hide]
    @inproceedings{sandbach2012binary,
        author = {G. Sandbach and S. Zafeiriou and M. Pantic},
        address = {Guildford, UK},
        booktitle = {Proceedings of the British Machine Vision Conference (BMVC 2012)},
        month = {September},
        title = {Binary Pattern Analysis for 3D Facial Action Unit Detection},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Binary Pattern Analysis for 3D Facial Action Unit Detection
    %A Sandbach, G.
    %A Zafeiriou, S.
    %A Pantic, M.
    %B Proceedings of the British Machine Vision Conference (BMVC 2012)
    %D 2012
    %8 September
    %C Guildford, UK
    %F sandbach2012binary

  5. Action Unit detection using sparse appearance descriptors in space-time video volumes

    B. Jiang, M. F. Valstar, M. Pantic. Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG'11). Santa Barbara, CA, USA, pp. 314 - 321, March 2011.

    Bibtex reference [hide]
    @inproceedings{BihanEtAlFG2011,
        author = {B. Jiang and M. F. Valstar and M. Pantic},
        pages = {314--321},
        address = {Santa Barbara, CA, USA},
        booktitle = {Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG'11)},
        month = {March},
        title = {Action Unit detection using sparse appearance descriptors in space-time video volumes},
        year = {2011},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Action Unit detection using sparse appearance descriptors in space-time video volumes
    %A Jiang, B.
    %A Valstar, M. F.
    %A Pantic, M.
    %B Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG?11)
    %D 2011
    %8 March
    %C Santa Barbara, CA, USA
    %F BihanEtAlFG2011
    %P 314-321

  6. A dynamic texture based approach to recognition of facial actions and their temporal models

    S. Koelstra, M. Pantic, I. Patras. IEEE Transactions on Pattern Analysis and Machine Intelligence. 32(11): pp. 1940 - 1954, november 2010.

    Bibtex reference [hide]
    @article{Koelstra2010adtba,
        author = {S. Koelstra and M. Pantic and I. Patras},
        pages = {1940--1954},
        journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
        month = {november},
        number = {11},
        title = {A dynamic texture based approach to recognition of facial actions and their temporal models},
        volume = {32},
        year = {2010},
    }
    Endnote reference [hide]
    %0 Journal Article
    %T A dynamic texture based approach to recognition of facial actions and their temporal models
    %A Koelstra, S.
    %A Pantic, M.
    %A Patras, I.
    %J IEEE Transactions on Pattern Analysis and Machine Intelligence
    %D 2010
    %8 november
    %V 32
    %N 11
    %F Koelstra2010adtba
    %P 1940-1954

  7. Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics

    S. Koelstra, M. Pantic. Proceedings of IEEE Int'l Conf. Automatic Face and Gesture Recognition (FG'08), Amsterdam, Netherlands. pp. 1 - 8, September 2008.

    Bibtex reference [hide]
    @inproceedings{Koelstra2008nrufd,
        author = {S. Koelstra and M. Pantic},
        pages = {1--8},
        booktitle = {Proceedings of IEEE Int'l Conf. Automatic Face and Gesture Recognition (FG'08), Amsterdam, Netherlands},
        month = {September},
        title = {Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics},
        year = {2008},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Non-rigid registration using free-form deformations for recognition of facial actions and their temporal dynamics
    %A Koelstra, S.
    %A Pantic, M.
    %B Proceedings of IEEE Int?l Conf. Automatic Face and Gesture Recognition (FG?08), Amsterdam, Netherlands
    %D 2008
    %8 September
    %F Koelstra2008nrufd
    %P 1-8

  8. Combined Support Vector Machines and Hidden Markov Models for Modeling Facial Action Temporal Dynamics

    M. F. Valstar, M. Pantic. IEEE Int'l Conf. Computer Vision (ICCV'07), Workshop on Human Computer Interaction (HCI'07). Rio de Janeiro, Brazil, pp. 118 - 127, October 2007.

    Bibtex reference [hide]
    @inproceedings{Valstar2007csvma,
        author = {M. F. Valstar and M. Pantic},
        pages = {118--127},
        address = {Rio de Janeiro, Brazil},
        booktitle = {IEEE Int'l Conf. Computer Vision (ICCV'07), Workshop on Human Computer Interaction (HCI'07)},
        month = {October},
        title = {Combined Support Vector Machines and Hidden Markov Models for Modeling Facial Action Temporal Dynamics},
        year = {2007},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Combined Support Vector Machines and Hidden Markov Models for Modeling Facial Action Temporal Dynamics
    %A Valstar, M. F.
    %A Pantic, M.
    %B IEEE Int?l Conf. Computer Vision (ICCV?07), Workshop on Human Computer Interaction (HCI?07)
    %D 2007
    %8 October
    %C Rio de Janeiro, Brazil
    %F Valstar2007csvma
    %P 118-127

  9. Dynamics of Facial Expression: Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences

    M. Pantic, I. Patras. IEEE Transactions on Systems, Man and Cybernetics - Part B. 36(2): pp. 433 - 449, 2006.

    Bibtex reference [hide]
    @article{Pantic2006dofer,
        author = {M. Pantic and I. Patras},
        pages = {433--449},
        journal = {IEEE Transactions on Systems, Man and Cybernetics - Part B},
        number = {2},
        title = {Dynamics of Facial Expression: Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences},
        volume = {36},
        year = {2006},
    }
    Endnote reference [hide]
    %0 Journal Article
    %T Dynamics of Facial Expression: Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences
    %A Pantic, M.
    %A Patras, I.
    %J IEEE Transactions on Systems, Man and Cybernetics - Part B
    %D 2006
    %V 36
    %N 2
    %F Pantic2006dofer
    %P 433-449

  10. Fully automatic facial action unit detection and temporal analysis

    M. F. Valstar, M. Pantic. Proceedings of IEEE Int'l Conf. Computer Vision and Pattern Recognition (CVPR'06). New York, USA, 3: pp. 149, June 2006.

    Bibtex reference [hide]
    @inproceedings{Valstar2006fafau,
        author = {M. F. Valstar and M. Pantic},
        pages = {149},
        address = {New York, USA},
        booktitle = {Proceedings of IEEE Int'l Conf. Computer Vision and Pattern Recognition (CVPR'06)},
        month = {June},
        title = {Fully automatic facial action unit detection and temporal analysis},
        volume = {3},
        year = {2006},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Fully automatic facial action unit detection and temporal analysis
    %A Valstar, M. F.
    %A Pantic, M.
    %B Proceedings of IEEE Int?l Conf. Computer Vision and Pattern Recognition (CVPR?06)
    %D 2006
    %8 June
    %V 3
    %C New York, USA
    %F Valstar2006fafau
    %P 149

  11. Detecting facial actions and their temporal segments in nearly frontal-view face image sequences

    M. Pantic, I. Patras. Proceedings of IEEE Int'l Conf. Systems, Man and Cybernetics (SMC'05). Waikoloa, Hawaii, pp. 3358 - 3363, October 2005.

    Bibtex reference [hide]
    @inproceedings{Pantic2005dfaat,
        author = {M. Pantic and I. Patras},
        pages = {3358--3363},
        address = {Waikoloa, Hawaii},
        booktitle = {Proceedings of IEEE Int'l Conf. Systems, Man and Cybernetics (SMC'05)},
        month = {October},
        title = {Detecting facial actions and their temporal segments in nearly frontal-view face image sequences},
        year = {2005},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Detecting facial actions and their temporal segments in nearly frontal-view face image sequences
    %A Pantic, M.
    %A Patras, I.
    %B Proceedings of IEEE Int?l Conf. Systems, Man and Cybernetics (SMC?05)
    %D 2005
    %8 October
    %C Waikoloa, Hawaii
    %F Pantic2005dfaat
    %P 3358-3363

  12. Facial Action Unit Detection using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data

    M. F. Valstar, I. Patras, M. Pantic. Proceedings of IEEE Int'l Conf. Computer Vision and Pattern Recognition (CVPR-W'05). San Diego, USA, 3: pp. 76, June 2005.

    Bibtex reference [hide]
    @inproceedings{Valstar2005faudu,
        author = {M. F. Valstar and I. Patras and M. Pantic},
        pages = {76},
        address = {San Diego, USA},
        booktitle = {Proceedings of IEEE Int'l Conf. Computer Vision and Pattern Recognition (CVPR-W'05)},
        month = {June},
        title = {Facial Action Unit Detection using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data},
        volume = {3},
        year = {2005},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Facial Action Unit Detection using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data
    %A Valstar, M. F.
    %A Patras, I.
    %A Pantic, M.
    %B Proceedings of IEEE Int?l Conf. Computer Vision and Pattern Recognition (CVPR-W?05)
    %D 2005
    %8 June
    %V 3
    %C San Diego, USA
    %F Valstar2005faudu
    %P 76

  13. Motion History for Facial Action Detection from Face Video

    M. F. Valstar, M. Pantic, I. Patras. Proceedings of IEEE Int'l Conf. Systems, Man and Cybernetics (SMC'04). The Hague, Netherlands, pp. 635 - 640, October 2004.

    Bibtex reference [hide]
    @inproceedings{Valstar2004mhffa,
        author = {M. F. Valstar and M. Pantic and I. Patras},
        pages = {635--640},
        address = {The Hague, Netherlands},
        booktitle = {Proceedings of IEEE Int'l Conf. Systems, Man and Cybernetics (SMC'04)},
        month = {October},
        title = {Motion History for Facial Action Detection from Face Video},
        year = {2004},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Motion History for Facial Action Detection from Face Video
    %A Valstar, M. F.
    %A Pantic, M.
    %A Patras, I.
    %B Proceedings of IEEE Int?l Conf. Systems, Man and Cybernetics (SMC?04)
    %D 2004
    %8 October
    %C The Hague, Netherlands
    %F Valstar2004mhffa
    %P 635-640

  14. Facial Action Recognition for Facial Expression Analysis from Static Face Images

    M. Pantic, L. Rothkrantz. IEEE Transactions on Systems, Man and Cybernetics - Part B. 34(3): pp. 1449 - 1461, 2004.

    Bibtex reference [hide]
    @article{Pantic2004farff,
        author = {M. Pantic and L. Rothkrantz},
        pages = {1449--1461},
        journal = {IEEE Transactions on Systems, Man and Cybernetics - Part B},
        number = {3},
        title = {Facial Action Recognition for Facial Expression Analysis from Static Face Images},
        volume = {34},
        year = {2004},
    }
    Endnote reference [hide]
    %0 Journal Article
    %T Facial Action Recognition for Facial Expression Analysis from Static Face Images
    %A Pantic, M.
    %A Rothkrantz, L.
    %J IEEE Transactions on Systems, Man and Cybernetics - Part B
    %D 2004
    %V 34
    %N 3
    %F Pantic2004farff
    %P 1449-1461

  15. Facial Action Unit Recognition using Temporal Templates

    M. F. Valstar, I. Patras, M. Pantic. Proceedings of IEEE Int'l Workshop on Robot-Human Interaction (RO-MAN'04). Kurashiki, Japan, pp. 253 - 258, September 2004.

    Bibtex reference [hide]
    @inproceedings{Valstar2004fauru,
        author = {M. F. Valstar and I. Patras and M. Pantic},
        pages = {253--258},
        address = {Kurashiki, Japan},
        booktitle = {Proceedings of IEEE Int'l Workshop on Robot-Human Interaction (RO-MAN'04)},
        month = {September},
        title = {Facial Action Unit Recognition using Temporal Templates},
        year = {2004},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Facial Action Unit Recognition using Temporal Templates
    %A Valstar, M. F.
    %A Patras, I.
    %A Pantic, M.
    %B Proceedings of IEEE Int?l Workshop on Robot-Human Interaction (RO-MAN?04)
    %D 2004
    %8 September
    %C Kurashiki, Japan
    %F Valstar2004fauru
    %P 253-258

  16. Facial Action Detection from Dual-View Static Face Images

    M. Pantic, L. Rothkrantz. Proceedings of IEEE Int'l Conf. Fuzzy Systems (IEEE-Fuzz'04). Budapest, Hungary, 1: pp. 39 - 44, July 2004.

    Bibtex reference [hide]
    @inproceedings{Pantic2004fadfd,
        author = {M. Pantic and L. Rothkrantz},
        pages = {39--44},
        address = {Budapest, Hungary},
        booktitle = {Proceedings of IEEE Int'l Conf. Fuzzy Systems (IEEE-Fuzz'04)},
        month = {July},
        title = {Facial Action Detection from Dual-View Static Face Images},
        volume = {1},
        year = {2004},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Facial Action Detection from Dual-View Static Face Images
    %A Pantic, M.
    %A Rothkrantz, L.
    %B Proceedings of IEEE Int?l Conf. Fuzzy Systems (IEEE-Fuzz?04)
    %D 2004
    %8 July
    %V 1
    %C Budapest, Hungary
    %F Pantic2004fadfd
    %P 39-44

  17. Temporal modeling of facial actions from face profile image sequences

    M. Pantic, I. Patras. Proceedings of IEEE Int'l Conf. Multimedia and Expo (ICME'04). Taipei, Taiwan, 1: pp. 49 - 52, June 2004.

    Bibtex reference [hide]
    @inproceedings{Pantic2004tmofa,
        author = {M. Pantic and I. Patras},
        pages = {49--52},
        address = {Taipei, Taiwan},
        booktitle = {Proceedings of IEEE Int'l Conf. Multimedia and Expo (ICME'04)},
        month = {June},
        title = {Temporal modeling of facial actions from face profile image sequences},
        volume = {1},
        year = {2004},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Temporal modeling of facial actions from face profile image sequences
    %A Pantic, M.
    %A Patras, I.
    %B Proceedings of IEEE Int?l Conf. Multimedia and Expo (ICME?04)
    %D 2004
    %8 June
    %V 1
    %C Taipei, Taiwan
    %F Pantic2004tmofa
    %P 49-52

  18. Facial action recognition in face profile image sequences

    M. Pantic, I. Patras, L. Rothkrantz. Proceedings of IEEE Int'l Conf. Multimedia and Expo (ICME'02). Lausanne, Switzerland, 1: pp. 37 - 40, August 2002.

    Bibtex reference [hide]
    @inproceedings{Pantic2002farif,
        author = {M. Pantic and I. Patras and L. Rothkrantz},
        pages = {37--40},
        address = {Lausanne, Switzerland},
        booktitle = {Proceedings of IEEE Int'l Conf. Multimedia and Expo (ICME'02)},
        month = {August},
        title = {Facial action recognition in face profile image sequences},
        volume = {1},
        year = {2002},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Facial action recognition in face profile image sequences
    %A Pantic, M.
    %A Patras, I.
    %A Rothkrantz, L.
    %B Proceedings of IEEE Int?l Conf. Multimedia and Expo (ICME?02)
    %D 2002
    %8 August
    %V 1
    %C Lausanne, Switzerland
    %F Pantic2002farif
    %P 37-40

  19. Expert System for Automatic Analysis of Facial Expressions

    M. Pantic, L. Rothkrantz. Image and Vision Computing Journal. 18(11): pp. 881 - 905, 2000.

    Bibtex reference [hide]
    @article{Pantic2000esfaa,
        author = {M. Pantic and L. Rothkrantz},
        pages = {881--905},
        journal = {Image and Vision Computing Journal},
        number = {11},
        title = {Expert System for Automatic Analysis of Facial Expressions},
        volume = {18},
        year = {2000},
    }
    Endnote reference [hide]
    %0 Journal Article
    %T Expert System for Automatic Analysis of Facial Expressions
    %A Pantic, M.
    %A Rothkrantz, L.
    %J Image and Vision Computing Journal
    %D 2000
    %V 18
    %N 11
    %F Pantic2000esfaa
    %P 881-905

  20. An Expert System for Recognition of Facial Actions and Their Intensity

    M. Pantic, L. Rothkrantz. Proceedings of AAAI Int'l Conf. Applications of AI (IAAI'00). Austin, USA, pp. 1026 - 1033, August 2000.

    Bibtex reference [hide]
    @inproceedings{Pantic2000aesfr,
        author = {M. Pantic and L. Rothkrantz},
        pages = {1026--1033},
        address = {Austin, USA},
        booktitle = {Proceedings of AAAI Int'l Conf. Applications of AI (IAAI'00)},
        month = {August},
        title = {An Expert System for Recognition of Facial Actions and Their Intensity},
        year = {2000},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T An Expert System for Recognition of Facial Actions and Their Intensity
    %A Pantic, M.
    %A Rothkrantz, L.
    %B Proceedings of AAAI Int?l Conf. Applications of AI (IAAI?00)
    %D 2000
    %8 August
    %C Austin, USA
    %F Pantic2000aesfr
    %P 1026-1033

  21. Automation of Non-verbal Communication of Facial Expressions

    M. Pantic, L. Rothkrantz, H. Koppelaar. Proceedings of European Conf. Multimedia (EUROMEDIA '98). Leicester, UK, pp. 86 - 93, January 1998.

    Bibtex reference [hide]
    @inproceedings{Pantic1998aonco,
        author = {M. Pantic and L. Rothkrantz and H. Koppelaar},
        pages = {86--93},
        address = {Leicester, UK},
        booktitle = {Proceedings of European Conf. Multimedia (EUROMEDIA '98)},
        month = {January},
        title = {Automation of Non-verbal Communication of Facial Expressions},
        year = {1998},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Automation of Non-verbal Communication of Facial Expressions
    %A Pantic, M.
    %A Rothkrantz, L.
    %A Koppelaar, H.
    %B Proceedings of European Conf. Multimedia (EUROMEDIA ?98)
    %D 1998
    %8 January
    %C Leicester, UK
    %F Pantic1998aonco
    %P 86-93