Face deidentification with generative deep models (FaceGEN)

About

The research project “Face deidentification with Generative Deep Models” (FaceGEN), strives to conduct research on deidentification technology and visual privacy with a particular focus on deep learning, which has recently been shown to be a highly effective tool for various computer vision and machine learning problems. Our goal is to develop deep generative models and conditional face synthesis techniques that can be used for deidentification and privacy protection with still images, but also with video, where multiple faces in cluttered and unconstrained scenes may appear in the data. The main tangible results of the project will be novel generative deep models and input-conditioned image synthesis techniques that are able to deidentify all parts of the facial data photo-realistically and ensure higher levels of privacy for users.

FaceGEN (ARRS J2-1734) is a fundamental research project funded by the Slovenian Research Agency (ARRS) in in the period: 1.7.2019 – 31.6.2022 (1,66 FTE per year).

The Principal Investigator (PI) of FaceGEN is Assoc. Prof. Vitomir Štruc, PhD.

Link to SICRIS: Follow me.

Project overview

FaceGEN is structured into 7 work packages:

WP1: Project management
WP2: Prerequisites
WP3: Face synthesis with generative models
WP4: Face alteration with gradient ascent
WP5: Face deidentification in video data
WP6: Demo room and exploitation
WP7: Dissemination

The R&D work on this work packages is expected to result in:

Formal privacy schemes based on deep models
Photo-realistic face synthesis techniques
Novel deidentification methods for image and video data

Project phases

Year 1: Activities on work packages WP1, WP2, WP3, WP4, WP7
Year 2: Activities on work packages WP1, WP3, WP4, WP7
Year 3: Activities on work packages WP1, WP5, WP6, WP7

Partners

FaceGEN is conducted jointly by:

Participating researchers

Assoc. Prof. Vitomir Štruc, PhD – Project Leader (PI)
Assoc. Prof. Simon Dobrišek, PhD, Researcher LMI
Ass. Prof. Janez Perš, PhD, Researcher LMI
Ass. Janez Križaj, PhD, Resercher LMI
Ass. Klemen Grm, Researcher LMI
Ass. Martin Pernuš, Researcher LMI (PhD Student)
Peter Rot, Researcher LMI (PhD Student)
Prof. Peter Peer, PhD, PI at CVL
Prof. Franc Solina, PhD, Researcher CVL
Ass. Borut Batagelj, PhD, Researcher CVL
Ass. Žiga Emeršič, Researcher CVL (PhD Student)
Jerneja Žganec Gros, PhD, PI at Alpineon
Mario Žganec, PhD, Researcher Alpineon
Robert Sedevčič, Technical staff (participation 2019)

International Advisory Committee

Prof. Arun Ross, PhD, Michigan State University
Prof. Slobodan Ribarić, PhD, University of Zagreb
Ass. Prof. Walter Scheirer, PhD, University of Notre Dame

Project publications

B. Meden, M. Gonzales-Hernandez, P. Peer, V. Štruc, Face Deidentification with Controllable Privacy Protection, Image and Vision Computing (SCI IF 2021: 3.860), 2023 [PDF].
P. Rot, P. Peer, V. Štruc, PrivacyProber: Assessment and Detection of Soft-Biometric Privacy-Enhancing Techniques, arXiV 2022, Under review at TDSC [PDF]
R. Tolosana, C. Rathgeb, R. Vera-Rodriguez, C. Busch, L. Verdilova, S. Lyu, H.H. Nguyen, J. Yamagishi, I. Echizen, P. Rot, K. Grm, V. Štruc, A. Dantcheva, Z. Akhtar, S. Romero-Tapiador, J. Fierrez, A. Morales, J. Ortega-Garcia, E. Kindt, C. Jasserand, T. Kalvet, M. Tiits, Future Trends in Digital Face Manipulation and Detection, in: C. Rathgeb, R. Tulsana, R. Vera-Rodriguez, C. Busch (Eds.), Hanbook of Digital Face Manipulation and Detection, Springer, pp. 464-482, 2022 [PDF]
Peter Rot, Peter Peer, Vitomir Štruc, Detecting Soft-Biometric Privacy Enhancement, in: C. Rathgeb, R. Tulsana, R. Vera-Rodriguez, C. Busch (Eds.), Hanbook of Digital Face Manipulation and Detection, Springer, pp. 391-412, 2022 [PDF]
Dailé Osorio-Roig, Christian Rathgeb, Pawel Drozdowski, Philipp Terhörst, Vitomir Štruc, Christoph Busch, An Attack on Feature Level-based Facial Soft-biometric Privacy Enhancement, IEEE Transactions on Biometrics, Identity and Behavior (TBIOM), vol. 4, iss. 2, pp. 263-275, 2022 [PDF]
Peter Rot, Peter Peer, Vitomir Štruc, Detekcija manipulacij za ohranjanje zasebnosti mehkih atributov na slikah obraza, Uporabna informatika, letn. 29, št. 4, str. 214-219, 2021 [PDF]
Blaž Meden, Peter Rot, Philipp Terhorst, Naser Damer, Arjan Kuijper, Arun Ross, Walter Scheirer, Peter Peer, Vitomir Štruc, Privacy-Enhancing Face Biometrics: A Comprehensive Survey, IEEE Transactions on Information Forensics and Security (SCI IF 2021: 7.231), 2021 [PDF]
Blaz Bortolato, Marija Ivanovska, Peter Rot, Janez Krizaj, Philipp Terhörst, Naser Damer, Peter Peer, Vitomir Štruc, Learning privacy-enhancing face representations through feature disentanglement, In: Proceedings of FG 2020, Buenos Aires, Argentina, 2020 [PDF].
Philipp Terhörst, Kevin Riehl, Naser Damer, Peter Rot, Blaž Bortolato, Florian Kirchbuchner, Vitomir Štruc, Arjan Kuijper, PE-MIU: A Training-Free Privacy-Enhancing Face Recognition Approach Based on Minimum Information Units. IEEE Access (SCI IF 2019: 3.745), 2020 [PDF].
Philipp Terhörst, Marco Huber, Naser Damer, Peter Rot, Florian Kirchbuchner, Vitomir Štruc, Arjan Kuijper, Privacy Evaluation Protocols for the Evaluation of Soft-Biometric Privacy-Enhancing Technologies, In: Proceedings of BIOSIG 2020, Darmstadt, Germany, 2020 [PDF].

Related publications

Matej Vitek; Abhijit Das; Diego Rafael Lucio; Luiz Antonio Zanlorensi Jr.; David Menotti; Jalil Nourmohammadi Khiarak; Mohsen Akbari Shahpar; Meysam Asgari-Chenaghlu; Farhang Jaryani; Juan E. Tapia; Andres Valenzuela; Caiyong Wang; Yunlong Wang; Zhaofeng He; Zhenan Sun; Fadi Boutros; Naser Damer; Jonas Henry Grebe; Arjan Kuijper; Kiran Raja; Gourav Gupta; Georgios Zampoukis; Lazaros Tsochatzidis; Ioannis Pratikakis; S. V. Aruna Kumar; B. S. Harish; Umapada Pal; Peter Peer; Vitomir Štruc, Exploring Bias in Sclera Segmentation Models: A Group Evaluation Approach, IEEE Transactions on Information Forensics and Security (SCI IF 2020: 7.178), vol. 18, pp. 190-205, 2023 [PDF]
HUBER, Marco, BOUTROS, Fadi, LUU, Anh Thi, RAJA, Kiran, RAMACHANDRA, Raghavendra, DAMER, Naser, NETO, Pedro C., GONÇALVES, Tiago, SEQUEIRA, Ana F., CARDOSO, Jaime S., TREMOÇO, João, LOURENÇO, Miguel, SERRA, Sergio, CERMEÑO, Eduardo, IVANOVSKA, Marija, BATAGELJ, Borut, KRONOVŠEK, Andrej, PEER, Peter, ŠTRUC, Vitomir. SYN-MAD 2022 : competition on face morphing attack detection based on privacy-aware synthetic training data. In: IEEE International Joint Conference on Biometrics, 2022 [PDF]
BABNIK, Žiga, ŠTRUC, Vitomir. Assessing bias in face image quality assessment. V: Proceedings : 30th European Signal Processing Conference (EUSIPCO 2022), 2022 [PDF]

Patent

ŠTRUC, Vitomir, IVANOVSKA, Marija, ROT, Peter, PEER, Peter, ČERNE, Tomaž, ŽGANEC GROS, Jerneja. Postopek za prepoznavanje identitete na podlagi obraznih lastnosti z upoštevanjem varovanja zasebnosti : patent SI 25987 A, 2021-10-29. Ljubljana: Urad RS za intelektualno lastnino, 2021.

Master Theses

Žiga Babnik, Face Deidentification using Face Swapping, Master Thesis, Co-supervisors: Peter Peer and Vitomir Štruc, 2021 [PDF]

Nejc Sušin, Improving a deidentification model using generative adversarial networks, Master Thesis, Co-supervisors: Peter Peer and Vitomir Štruc, 2019 [PDF]

Invited Talks

ŠTRUC, Vitomir. Generative models in computer vision : from fashion and beauty to biometrics : the 1st Workshop on Interdisciplinary Applications of Biometrics and Identity Science, January 5th 2023, Waikoloa Beach, Hawaii.
ŠTRUC, Vitomir. DeepFake – kaj je to in zakaj je pomembno? : 11. mednarodna konferenca IIA – Slovenskega inštituta , Bled, 26. 5. 2022
ŠTRUC, Vitomir. Generative models in computer vision : from fashion and beauty to biometrics : Queensland University of Technology, 22 December 2022.
ŠTRUC, Vitomir. Generative models in biometrics : Trustworthy Biometrics Webinar, IEEE Biometrics Beijing Chapter, 6th May 2022
ŠTRUC, Vitomir. Generative models in computer vision and biometrics : Istanbul Technical University, Turkey, 21. 9. 2022.
ŠTRUC, Vitomir. Generative models in computer vision and biometrics : EURASIP JIVP Webinar, 6 October 2022
ŠTRUC, Vitomir. Generative models in computer vision and biometrics : University of Salzburg, Austria, 14 June 2022.
ŠTRUC, Vitomir. Photorealistic face editing via latent code optimization : Workshop Digital Face Manipulation & Detection, organized by European Association for Biometrics (EAB), 12. Jul. 2022.
Peter Peer, Overview of privacy-enhancing face biometrics, Keynote Talk, ELMAR2021: 63rd International Symposium ELMAR-2021, Zadar, Croatia, 13-15 September 2021. Zadar: Croatian Society Electronics in Marine, 2021.
Peter Peer, Generative deep neural networks for face deidentification, Keynote Talk, IC3A 2020.
Peter Peer, From sclera recognition to face biometrics privacy-enhancing techniques and soft-biometric modalities, lecture at Silesian University of Technology, Gliwice, Poland, 13. 10. 2021.
Peter Peer, Research conducted in the Computer Vision Laboratory with focus on biometrics, lecture at Galgotias University, New Delhi, India, 3rd February 2020.

TV Shows

DACINGER, Renata (voditelj oddaje, oseba, ki intervjuva), IVANUŠ ČUČEK, Nataša (oseba, ki intervjuva, scenarist), ŠTRUC, Vitomir (intervjuvanec), BATAGELJ, Borut (intervjuvanec), IVANOVSKA, Marija (intervjuvanec), TOMANIĆ TRIVUNDŽA, Ilija (intervjuvanec). Ponarejeni videoposnetki. Ljubljana: Radiotelevizija Slovenija javni zavod, Ugriznimo znanost, 2022 [Link].
ŠTRUC, Vitomir (intervjuvanec), JONTES, Dejan (intervjuvanec), KALUŽA, Jernej (intervjuvanec), MEDEN, Blaž (intervjuvanec). Ponarejeni videi = Deepfake video. Ljubljana: Radiotelevizija Slovenija, Na kratko, 2020.

Deliverables

Some of the project deliverables represent reports, which (in the first stage) are made publicly available to the funding agency only. The deliverables will be used as the basis for the project publications and will be released to the general public after publication in a peer reviewed venue. The deliverables can be accessed from the FaceGEN Deliverable page.