Implicit, human-behaviour-based tagging

In multimedia management and sharing, tags assigned to the content play an important role in search and retrieval. With the help of tags, multimedia contents, such as, images, videos and web pages, can be efficiently classified, indexed and searched. The tagging technique currently adopted by most social media websites is explicit tagging, in which images are annotated based on the image content by human, either experts or amateurs. Although explicit image tagging has been successfully applied in a wide range of WEB 2.0 applications these days, it still has several drawbacks. First, the tags obtained by explicit tagging are likely to be inaccurate in practice. When tagging, people usually behave according to their individual interpretation of the content, personal and social needs without considering whether the tags can be used to improve the performance of retrieval. Second, it requires extra effort from users. Explicit tagging is sometimes really inconvenient for users since adding tags to images may take a large amount of time and energy.
 
In contrast to explicit tagging, implicit tagging is the technique to annotate multimedia data based on user's nonverbal reactions, such as facial expression and head gesture. The data is tagged in a implicit way because it is based on user's reaction without explicitly requesting a user to associate tag with the data. One possible application of implicit tagging is to assess the correctness of explicit tags. For example, if a user smiles or nodes the head while watching a tagged datum, this may imply that the tag is correct; however, if the user displays an expression of surprise or disappointment, the tag is likely to be incorrect. Another possible application of implicit tagging is to assign new explicit tags. For instance, if the user laughs while watching an untagged datum, it is likely that it contains some funny elements, thus can be labelled as "funny" or "interesting". Our current research focuses on using user's facial expressino and eye movement to predict the correctness of tags.

Involved group members

Maja Pantic, Konstantinos Bousmalis, Sebastian Kaltwang, Mohammad Soleymani, Jun Jiao

Related Publications

  1. Human-Centered Implicit Tagging: Overview and Perspectives

    M. Soleymani, M. Pantic. IEEE International Conference on Systems, Man and Cybernetics, 2012. IEEE, IEEE Press, pp. 3304 - 3309, October 2012.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{smc2012_implicit,
        author = {M. Soleymani and M. Pantic},
        pages = {3304--3309},
        booktitle = { IEEE International Conference on Systems, Man and Cybernetics, 2012},
        month = {October},
        organization = {IEEE},
        publisher = {IEEE Press},
        title = {Human-Centered Implicit Tagging: Overview and Perspectives},
        year = {2012},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Human-Centered Implicit Tagging: Overview and Perspectives
    %A Soleymani, M.
    %A Pantic, M.
    %B IEEE International Conference on Systems, Man and Cybernetics, 2012
    %D 2012
    %8 October
    %I IEEE Press
    %F smc2012_implicit
    %P 3304-3309

  2. Implicit Image Tagging via Facial Information

    J. Jiao, M. Pantic. Proceedings ACM Int'l Workshop on Social Signal Processing (SSPW'10), Firenze, Italy. October 2010.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{Jiao2010iitvf,
        author = {J. Jiao and M. Pantic},
        booktitle = {Proceedings ACM Int'l Workshop on Social Signal Processing (SSPW'10), Firenze, Italy},
        month = {October},
        title = {Implicit Image Tagging via Facial Information},
        year = {2010},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Implicit Image Tagging via Facial Information
    %A Jiao, J.
    %A Pantic, M.
    %B Proceedings ACM Int?l Workshop on Social Signal Processing (SSPW?10), Firenze, Italy
    %D 2010
    %8 October
    %F Jiao2010iitvf

  3. Implicit Human-Centered Tagging

    M. Pantic, A. Vinciarelli. IEEE Signal Processing Magazine. 26(6): pp. 173 - 180, 2009.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @article{Pantic2009iht,
        author = {M. Pantic and A. Vinciarelli},
        pages = {173--180},
        journal = {IEEE Signal Processing Magazine},
        number = {6},
        title = {Implicit Human-Centered Tagging},
        volume = {26},
        year = {2009},
    }
    Endnote reference [hide]
    %0 Journal Article
    %T Implicit Human-Centered Tagging
    %A Pantic, M.
    %A Vinciarelli, A.
    %J IEEE Signal Processing Magazine
    %D 2009
    %V 26
    %N 6
    %F Pantic2009iht
    %P 173-180

  4. Implicit Human-Centered Tagging

    A. Vinciarelli, N. Suditu, M. Pantic. Proceedings of IEEE Int'l Conf. Multimedia, Expo (ICME'09). Cancun, Mexico, pp. 1428 - 1431, July 2009.

    [pdf] [bibtex] [endnote]
    Bibtex reference [hide]
    @inproceedings{Vinciarelli2009iht,
        author = {A. Vinciarelli and N. Suditu and M. Pantic},
        pages = {1428--1431},
        address = {Cancun, Mexico},
        booktitle = {Proceedings of IEEE Int'l Conf. Multimedia, Expo (ICME'09)},
        month = {July},
        title = {Implicit Human-Centered Tagging},
        year = {2009},
    }
    Endnote reference [hide]
    %0 Conference Proceedings
    %T Implicit Human-Centered Tagging
    %A Vinciarelli, A.
    %A Suditu, N.
    %A Pantic, M.
    %B Proceedings of IEEE Int?l Conf. Multimedia, Expo (ICME?09)
    %D 2009
    %8 July
    %C Cancun, Mexico
    %F Vinciarelli2009iht
    %P 1428-1431