Bermano, AH, Billeter, M orcid.org/0000-0003-1806-2587, Iwai, D et al. (1 more author) (2017) Makeup Lamps: Live Augmentation of Human Faces via Projection. Computer Graphics Forum, 36 (2). pp. 311-323. ISSN 0167-7055
Abstract
We propose the first system for live dynamic augmentation of human faces. Using projector‐based illumination, we alter the appearance of human performers during novel performances. The key challenge of live augmentation is latency — an image is generated according to a specific pose, but is displayed on a different facial configuration by the time it is projected. Therefore, our system aims at reducing latency during every step of the process, from capture, through processing, to projection. Using infrared illumination, an optically and computationally aligned high‐speed camera detects facial orientation as well as expression. The estimated expression blendshapes are mapped onto a lower dimensional space, and the facial motion and non‐rigid deformation are estimated, smoothed and predicted through adaptive Kalman filtering. Finally, the desired appearance is generated interpolating precomputed offset textures according to time, global position, and expression. We have evaluated our system through an optimized CPU and GPU prototype, and demonstrated successful low latency augmentation for different performers and performances with varying facial play and motion speed. In contrast to existing methods, the presented system is the first method which fully supports dynamic facial projection mapping without the requirement of any physical tracking markers and incorporates facial expressions.
Metadata
Item Type: | Article |
---|---|
Authors/Creators: |
|
Copyright, Publisher and Additional Information: | © 2017 The Author(s) Computer Graphics Forum © 2017 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd. This is the peer reviewed version of the following article: Bermano, A.H., Billeter, M., Iwai, D. and Grundhöfer, A. (2017), Makeup Lamps: Live Augmentation of Human Faces via Projection. Computer Graphics Forum, 36: 311-323. , which has been published in final form at https://doi.org/10.1111/cgf.13128. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived Versions. |
Keywords: | Categories and Subject Descriptors (according to ACM CCS); H.5.1 [HCI]: Multimedia Information Systems—Artificial, augmented, and virtual realities; I.3.7 [Computer Graphics]: Three‐Dimensional Graphics and Realism—Animation |
Dates: |
|
Institution: | The University of Leeds |
Academic Units: | The University of Leeds > Faculty of Engineering & Physical Sciences (Leeds) > School of Computing (Leeds) |
Depositing User: | Symplectic Publications |
Date Deposited: | 04 Jan 2021 14:07 |
Last Modified: | 04 Jan 2021 14:12 |
Status: | Published |
Publisher: | Wiley |
Identification Number: | 10.1111/cgf.13128 |
Related URLs: | |
Open Archives Initiative ID (OAI ID): | oai:eprints.whiterose.ac.uk:169265 |