Handling Facial Animation in Unity with iPhone X
Subscribe:  iCal  |  Google Calendar
7, Mar — 12, Jun
San Francisco US   19, Mar — 24, Mar
Anaheim US   23, Mar — 26, Mar
San Jose US   26, Mar — 30, Mar
Washington US   30, Mar — 2, Apr
Latest comments
by badminton rackets
21 min ago

Wonderful illustrated information. I thank you about that. No doubt it will be very useful for my future projects. Would like to see some other posts on the same subject! badminton rackets

by rayen
49 min ago

that's all nice but what's the purpose of that there is no consumer hardware that can't handle that in real game enviroment

by Romain
1 hours ago

Yeah I know normally a friendly artist is planning to make one about it

Handling Facial Animation in Unity with iPhone X
23 November, 2017

It appears that you can actually use iPhone X to deal with facial animation in Unity. Simeon Saëns and John Smith state that the ARKit face APIs are powerful enough to produce useful animation. And the thing is that you can capture it in real-time.

Face Demo

The developers managed to stream blend shape parameters live from iPhone X into Unity to control animation rig.

How does it work?

The demo consists of two parts. The iOS app and the Unity extension host.

iOS App

You can get it here.

The iOS app streams the Blend Shapes Apple provides in ARFaceAnchor.blendShapes to the Unity host through a UDP socket. Essentially emitting a stream of messages, each with 50 blend shapes in the format ‘blend-shape-name:blend-shape-value’.

Live Demo

There are lots of performance improvements to be made here but it works for the purpose of a demo.

Unity Extension Host

You can get it here.

Inside of the Unity host we have an extension which opens up a UDP socket to listen for the iPhone’s messages. When it receives a message it applies the blend shape values to the corresponding blend shape on the rig.

How to run the project

  • Clone and open the Unity project from here.
  • Run the Unity project’s scene
  • In the menu bar select iPhoneMoCap -> MeshPreview
  • Enable Mesh preview
  • Grab the iOS app project from here
  • Make sure your iPhone X is connected to the same Wifi network and build / run this application. (Don’t forget to pod install)
  • This application should discover the unity host and begin streaming the motion data.

Simeon Saëns and John Smith 

You can learn more about the project here.

Source: enabled.com

Leave a Reply

Be the First to Comment!