Understanding videos with face recognition: a complete pipeline and applications

Pasquale Lisena; Jorma Laaksonen; Raphaël Troncy

doi:10.1007/s00530-022-00959-x

Understanding videos with face recognition: a complete pipeline and applications

Pasquale Lisena^*
, Jorma Laaksonen
, Raphaël Troncy

^*Corresponding author for this work

EURECOM

Research output: Contribution to journal › Article › Scientific › peer-review

2 Citations (Scopus)

Abstract

When browsing or studying a video corpus, particularly relevant information consists in knowing who are the people appearing in the scenes. In this paper, we show how a combination of state of the art techniques can be organised in a pipeline for face recognition of celebrities. In particular, we propose a system which combines MTCNN for detecting faces and FaceNet for extracting face embeddings, which are used to train a set of classifiers. The face recognition results obtained at a frame level are then combined with those in consecutive frames, relying on automatic object tracking. Differently from previous work, we use images automatically retrieved by web search engines. We evaluate the systems one three datasets including historical videos from 1945 to 1969 and contemporary videos, obtaining a good precision score. In addition, we show how the obtained results can be applied to foster historical studies.

Original language	English
Pages (from-to)	2147-2159
Number of pages	13
Journal	Multimedia Systems
Volume	28
Issue number	6
Early online date	15 Jun 2022
DOIs	https://doi.org/10.1007/s00530-022-00959-x
Publication status	Published - Dec 2022
MoE publication type	A1 Journal article-refereed

Funding

The authors would like to thank Bénédicte Pincemin for the valuable feedback, which helped to improve the paper. This work has been partially supported by the French National Research Agency (ANR) within the ANTRACT project (grant number ANR-17-CE38-0010) and by the Academy of Finland (project grants n. 329268 and 345791).

Keywords

Computer vision
Face recognition
Knowledge graphs
Web image retrieval

Access to Document

10.1007/s00530-022-00959-x

https://www.eurecom.fr/publication/6931

Cite this

@article{25f46defb2574fca927d28e51d6d0cfe,

title = "Understanding videos with face recognition: a complete pipeline and applications",

abstract = "When browsing or studying a video corpus, particularly relevant information consists in knowing who are the people appearing in the scenes. In this paper, we show how a combination of state of the art techniques can be organised in a pipeline for face recognition of celebrities. In particular, we propose a system which combines MTCNN for detecting faces and FaceNet for extracting face embeddings, which are used to train a set of classifiers. The face recognition results obtained at a frame level are then combined with those in consecutive frames, relying on automatic object tracking. Differently from previous work, we use images automatically retrieved by web search engines. We evaluate the systems one three datasets including historical videos from 1945 to 1969 and contemporary videos, obtaining a good precision score. In addition, we show how the obtained results can be applied to foster historical studies.",

keywords = "Computer vision, Face recognition, Knowledge graphs, Web image retrieval",

author = "Pasquale Lisena and Jorma Laaksonen and Rapha{\"e}l Troncy",

note = "Funding Information: The authors would like to thank B{\'e}n{\'e}dicte Pincemin for the valuable feedback, which helped to improve the paper. This work has been partially supported by the French National Research Agency (ANR) within the ANTRACT project (grant number ANR-17-CE38-0010) and by the Academy of Finland (project grants n. 329268 and 345791). Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.",

year = "2022",

month = dec,

doi = "10.1007/s00530-022-00959-x",

language = "English",

volume = "28",

pages = "2147--2159",

journal = "Multimedia Systems",

issn = "0942-4962",

publisher = "Springer",

number = "6",

}

TY - JOUR

T1 - Understanding videos with face recognition

T2 - a complete pipeline and applications

AU - Lisena, Pasquale

AU - Laaksonen, Jorma

AU - Troncy, Raphaël

N1 - Funding Information: The authors would like to thank Bénédicte Pincemin for the valuable feedback, which helped to improve the paper. This work has been partially supported by the French National Research Agency (ANR) within the ANTRACT project (grant number ANR-17-CE38-0010) and by the Academy of Finland (project grants n. 329268 and 345791). Publisher Copyright: © 2022, The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.

PY - 2022/12

Y1 - 2022/12

N2 - When browsing or studying a video corpus, particularly relevant information consists in knowing who are the people appearing in the scenes. In this paper, we show how a combination of state of the art techniques can be organised in a pipeline for face recognition of celebrities. In particular, we propose a system which combines MTCNN for detecting faces and FaceNet for extracting face embeddings, which are used to train a set of classifiers. The face recognition results obtained at a frame level are then combined with those in consecutive frames, relying on automatic object tracking. Differently from previous work, we use images automatically retrieved by web search engines. We evaluate the systems one three datasets including historical videos from 1945 to 1969 and contemporary videos, obtaining a good precision score. In addition, we show how the obtained results can be applied to foster historical studies.

AB - When browsing or studying a video corpus, particularly relevant information consists in knowing who are the people appearing in the scenes. In this paper, we show how a combination of state of the art techniques can be organised in a pipeline for face recognition of celebrities. In particular, we propose a system which combines MTCNN for detecting faces and FaceNet for extracting face embeddings, which are used to train a set of classifiers. The face recognition results obtained at a frame level are then combined with those in consecutive frames, relying on automatic object tracking. Differently from previous work, we use images automatically retrieved by web search engines. We evaluate the systems one three datasets including historical videos from 1945 to 1969 and contemporary videos, obtaining a good precision score. In addition, we show how the obtained results can be applied to foster historical studies.

KW - Computer vision

KW - Face recognition

KW - Knowledge graphs

KW - Web image retrieval

UR - https://www.scopus.com/pages/publications/85132123070

U2 - 10.1007/s00530-022-00959-x

DO - 10.1007/s00530-022-00959-x

M3 - Article

AN - SCOPUS:85132123070

SN - 0942-4962

VL - 28

SP - 2147

EP - 2159

JO - Multimedia Systems

JF - Multimedia Systems

IS - 6

ER -

Understanding videos with face recognition: a complete pipeline and applications

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

USSEE: Understanding speech and scene with ears and eyes (USSEE)

MoMaF: Movie Making Finland: Finnish fiction films as audiovisual big data, 19072017

MeMAD Laaksonen

Science-IT

Cite this

Understanding videos with face recognition: a complete pipeline and applications

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Projects

USSEE: Understanding speech and scene with ears and eyes (USSEE)

MoMaF: Movie Making Finland: Finnish fiction films as audiovisual big data, 19072017

MeMAD Laaksonen

Equipment

Science-IT

Cite this

MoMaF: Movie Making Finland: Finnish fiction films as audiovisual big data, 19072017