3D facial expression modelling and recognition

The lab is developing tools for static and dynamic 3D facial expression and action unit recognition.

Binary Pattern Based Features

Features based on Local Binary Patterns have been developed for analysis of 3D facial geometries for the purpose of facial action unit detection. They use the normals, depth map, or our proposed Azimuthal Projection Distance Image (APDI) as input. We then extract discriminative features via the LBP algorithm along with Gabor and monogenic filters, or by applying Local Phase Quantisation to these representations.

Dynamic Expression Analysis

A facial expression sequence is modelled by a four state model (neutral-onset-apex-offset). The method we developed uses free-form deformations to capture the 3D motion between frames. Vector projections of the motion in the sequence with space and time are produced and used to perform quad-tree decompositions for each of the onset and offset temporal segments in the expression (some examples of quadtree decompositions and the corresponding motion projections are depicted above). Feature extraction and classification can then be performed on each frame in the image sequence, and the results of this used to produce temporal models for the full expression [1].




Involved group members

Maja Pantic, Stylianos Ploumpis, James Booth, Shiyang Cheng, Baris Gecer, Dimitrios Kollias, Ioannis Marras, Georgia Rajamanoharan (Sandbach), Anastasios Roussos

Related Publications

  1. Static and Dynamic 3D Facial Expression Recognition: A Comprehensive Survey

    G. Sandbach, S. Zafeiriou, M. Pantic, L. Yin. Image and Vision Computing. 30(10): pp. 683 - 697, 3D Facial Behaviour Analysis and Understanding. 2012.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @article{sandbach2012survey,
        author = {G. Sandbach and S. Zafeiriou and M. Pantic and L. Yin},
        pages = {683--697},
        journal = {Image and Vision Computing},
        note = {3D Facial Behaviour Analysis and Understanding},
        number = {10},
        publisher = {Elsevier},
        title = {Static and Dynamic 3D Facial Expression Recognition: A Comprehensive Survey},
        url = {http://www.sciencedirect.com/science/article/pii/S0262885612000935},
        volume = {30},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Journal Article
    %T Static and Dynamic 3D Facial Expression Recognition: A Comprehensive Survey
    %A Sandbach, G.
    %A Zafeiriou, S.
    %A Pantic, M.
    %A Yin, L.
    %J Image and Vision Computing
    %D 2012
    %V 30
    %N 10
    %I Elsevier
    %F sandbach2012survey
    %O 3D Facial Behaviour Analysis and Understanding
    %U http://www.sciencedirect.com/science/article/pii/S0262885612000935
    %P 683-697

  2. Local Normal Binary Patterns for 3D Facial Action Unit Detection

    G. Sandbach, S. Zafeiriou, M. Pantic. Proceedings of the IEEE International Conference on Image Processing (ICIP 2012). Orlando, FL, USA, pp. 1813 - 1816, October 2012.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{sandbach2012normal,
        author = {G. Sandbach and S. Zafeiriou and M. Pantic},
        pages = {1813--1816},
        address = {Orlando, FL, USA},
        booktitle = {Proceedings of the IEEE International Conference on Image Processing (ICIP 2012)},
        month = {October},
        title = {Local Normal Binary Patterns for 3D Facial Action Unit Detection},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Local Normal Binary Patterns for 3D Facial Action Unit Detection
    %A Sandbach, G.
    %A Zafeiriou, S.
    %A Pantic, M.
    %B Proceedings of the IEEE International Conference on Image Processing (ICIP 2012)
    %D 2012
    %8 October
    %C Orlando, FL, USA
    %F sandbach2012normal
    %P 1813-1816

  3. Binary Pattern Analysis for 3D Facial Action Unit Detection

    G. Sandbach, S. Zafeiriou, M. Pantic. Proceedings of the British Machine Vision Conference (BMVC 2012). Guildford, UK, September 2012.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{sandbach2012binary,
        author = {G. Sandbach and S. Zafeiriou and M. Pantic},
        address = {Guildford, UK},
        booktitle = {Proceedings of the British Machine Vision Conference (BMVC 2012)},
        month = {September},
        title = {Binary Pattern Analysis for 3D Facial Action Unit Detection},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Binary Pattern Analysis for 3D Facial Action Unit Detection
    %A Sandbach, G.
    %A Zafeiriou, S.
    %A Pantic, M.
    %B Proceedings of the British Machine Vision Conference (BMVC 2012)
    %D 2012
    %8 September
    %C Guildford, UK
    %F sandbach2012binary

  4. Recognition of 3D Facial Expression Dynamics

    G. Sandbach, S. Zafeiriou, M. Pantic, D. Rueckert. Image and Vision Computing. 30(10): pp. 762 - 773, 3D Facial Behaviour Analysis and Understanding. 2012.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @article{sandbach2012recognition,
        author = {G. Sandbach and S. Zafeiriou and M. Pantic and D. Rueckert},
        pages = {762--773},
        journal = {Image and Vision Computing},
        note = {3D Facial Behaviour Analysis and Understanding},
        number = {10},
        title = {Recognition of 3D Facial Expression Dynamics},
        url = {http://www.sciencedirect.com/science/article/pii/S0262885612000157},
        volume = {30},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Journal Article
    %T Recognition of 3D Facial Expression Dynamics
    %A Sandbach, G.
    %A Zafeiriou, S.
    %A Pantic, M.
    %A Rueckert, D.
    %J Image and Vision Computing
    %D 2012
    %V 30
    %N 10
    %F sandbach2012recognition
    %O 3D Facial Behaviour Analysis and Understanding
    %U http://www.sciencedirect.com/science/article/pii/S0262885612000157
    %P 762-773

  5. A Dynamic Approach to the Recognition of 3D Facial Expressions and Their Temporal Models

    G. Sandbach, S. Zafeiriou, M. Pantic, D. Rueckert. Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG'11), Special Session: 3D Facial Behavior Analysis and Understanding. Santa Barbara, CA, USA, pp. 406 - 413, March 2011.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{GeorgiaEtAlFG2011,
        author = {G. Sandbach and S. Zafeiriou and M. Pantic and D. Rueckert},
        pages = {406--413},
        address = {Santa Barbara, CA, USA},
        booktitle = {Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG'11), Special Session: 3D Facial Behavior Analysis and Understanding},
        month = {March},
        title = {A Dynamic Approach to the Recognition of 3D Facial Expressions and Their Temporal Models},
        year = {2011},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T A Dynamic Approach to the Recognition of 3D Facial Expressions and Their Temporal Models
    %A Sandbach, G.
    %A Zafeiriou, S.
    %A Pantic, M.
    %A Rueckert, D.
    %B Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG?11), Special Session: 3D Facial Behavior Analysis and Understanding
    %D 2011
    %8 March
    %C Santa Barbara, CA, USA
    %F GeorgiaEtAlFG2011
    %P 406-413