Handling Facial Animation in Unity with iPhone X

It appears that you can actually use iPhone X to deal with facial animation in Unity.

It appears that you can actually use iPhone X to deal with facial animation in Unity. Simeon Saëns and John Smith state that the ARKit face APIs are powerful enough to produce useful animation. And the thing is that you can capture it in real-time.

Face Demo

The developers managed to stream blend shape parameters live from iPhone X into Unity to control animation rig.

How does it work?

The demo consists of two parts. The iOS app and the Unity extension host.

iOS App

You can get it here.

The iOS app streams the Blend Shapes Apple provides in ARFaceAnchor.blendShapes to the Unity host through a UDP socket. Essentially emitting a stream of messages, each with 50 blend shapes in the format ‘blend-shape-name:blend-shape-value’.

Live Demo

There are lots of performance improvements to be made here but it works for the purpose of a demo.

Unity Extension Host

You can get it here.

Inside of the Unity host we have an extension which opens up a UDP socket to listen for the iPhone’s messages. When it receives a message it applies the blend shape values to the corresponding blend shape on the rig.

How to run the project

Clone and open the Unity project from here.
Run the Unity project’s scene
In the menu bar select iPhoneMoCap -> MeshPreview
Enable Mesh preview
Grab the iOS app project from here
Make sure your iPhone X is connected to the same Wifi network and build / run this application. (Don’t forget to pod install)
This application should discover the unity host and begin streaming the motion data.

Simeon Saëns and John Smith

You can learn more about the project here.

You might also like

1
2
3
4
5

A Week After "Basically Announcing" Minecraft 2, Notch Basically Cancels It

Instead, he and his team will focus on the previously-announced retro-style roguelike.

+2

Discord Gets Sued Over Alleged Anti-Consumer Practices

The plaintiffs claim the platform has intentionally made it overly difficult to cancel subscriptions.

+1

News

Rumor: Possible Release Date for Grand Theft Auto 6 Revealed

A video game store from Uruguay appears to have disclosed the launch date for the gaming industry's most anticipated title.

+2

UE4 Master Materials
by Lincoln Hughes
Save yourself endless time and technical hurdles in the material editor with this pack of Prop Master Materials for UE4!
Error
(2000-0001)
Ornament Brushes/Alphas - Vol.2
by Jonas Ronnegard
An Ornament alpha/brush set for ZBrush, Substance Painter, Quixel DDO and NDO etc. 55 brushes and height/alpha maps, all 2048x2048 16bit in Tiff, Jpeg, PSD, Photoshop ABR brushes, as well as ZBrush Brushes.
Error
(2000-0001)