Smartphone videos produce highly realistic 3D face reconstructions
Smartphone videos produce highly realistic 3D face reconstructions
Epic Games acquires facial animation technology maker Cubic Motion
Epic Games acquires facial animation technology maker Cubic Motion
Quill Launches VR Animation and Storytelling Tools
Quill Launches VR Animation and Storytelling Tools
Facebook Launches ‘Quill 2.0’ with Enhanced Features Making It a “self-contained storytelling tool”
Facebook Launches ‘Quill 2.0’ with Enhanced Features Making It a “self-contained storytelling tool”
How Is Virtual Reality Impacting Video Marketing and Advertising?
How Is Virtual Reality Impacting Video Marketing and Advertising?

Audio- and Gaze-driven Facial Animation of Codec Avatars

Audio- and Gaze-driven Facial Animation of Codec Avatars

August 11, 2020

Codec Avatars are a recent class of learned, photorealistic face models that accurately represent the geometry and texture of a person in 3D (i.e., for virtual reality), and are almost indistinguishable from video. In this paper we describe the first approach to animate these parametric models in real-time which could be deployed on commodity virtual reality hardware using audio and/or eye tracking. Our goal is to display expressive conversations between individuals that exhibit important social signals such as laughter and excitement solely from latent cues in our lossy input signals. To this end we collected over 5 hours of high frame rate 3D face scans across three participants including traditional neutral speech as well as expressive and conversational speech. We investigate a multimodal fusion approach that dynamically identifies which sensor encoding should animate which parts of the face at any time. See the supplemental video which demonstrates our ability to generate full face motion far beyond the typically neutral lip articulations seen in competing work.

Techniques for Improved VR Video w/ John Carmack
Techniques for Improved VR Video w/ John Carmack
Virtual Reality headsets, gaming devices, cardboards and accessories
Virtual Reality headsets, gaming devices, cardboards and accessories
Augmenting Virtual Reality with Reality
Augmenting Virtual Reality with Reality
Skin machine enables physical pain in virtual reality for the first time
Skin machine enables physical pain in virtual reality for the first time