Dynamixyz team discussed its markerless facial mocap system, shared the technical details and possibilities of the system and mentioned the company's future plans.
About the Company
Dynamixyz stands as a leading facial animation technology provider for the Entertainment industry offering the widest and smartest range of markerless facial motion capture software on the market.
The company was founded in 2010 by PhDs in computer vision - among them Gaspard Breton CEO, Olivier Aubault General Manager, and Nicolas Stoiber, CTO. They first started by providing software but soon had to develop hardware as an enabler of the technology. Dynamixyz aims at delivering a first-class facial mocap experience. Our offer includes a complete software suite to capture facial motion for offline production purposes or real-time performances and a range of head-mounted camera systems tailored to any need. The company is celebrating 10 years of existence this April, and over the years our technology has been used by major international AAA video games studios as well as VR studios and world-acclaimed VFX houses, looking to boost productivity while maintaining a high level of quality in their pipeline.
To celebrate this, we have just unveiled a redesign of the company's visual identity with a new logo and website that we are excited to share with you.
On the Idea of the Facial Mocap Technology
10 years ago, most studios would go about facial motion capture using optical motion capture techniques available for the body: they would place numerous physical markers on the face and record their movements with an array of optical cameras in a motion capture volume. In addition to spending hours preparing actors and ensuring all markers were in place in the right position throughout the shoot, it required to own or rent a fully-equipped mocap stage in the first place.
At the time, Dynamixyz founders were working for other industries on non-intrusive measures of human facial movements without any markers, from simple video feeds.
They saw the benefit that professional facial motion capture could draw from such technologies. In 2010, they decided to found the company with a goal to make quality facial mocap accessible and usable by the greatest number of people, at a time when it only existed as a service for high-end studios. Bringing together a team of passionate and award-winning computer vision researchers and engineers, Dynamixyz became one of the very first companies to offer this technology as software that users could run on their own. It builds upon custom-made face tracking algorithms, able to track expressions from a simple video source, and delivering unrivaled high-quality results.
Dynamixyz' core software, Performer, learns from data - a collection of face snapshots of an actor recorded with a camera-specific expressive pattern of the actor based on the user’s annotations. As many machine learning technologies, once the system is trained, it can track any new video from the same actor automatically, making it possible to batch process massive amounts of shots.
Apart from being more comfortable for an actor, it also requires less hardware and fits tighter budgets. Anyone can animate a face with Performer technology even with a regular webcam. For productivity as well as data safety reasons, it is also important to us that our clients keep and process their data in-house. We do still provide support, obviously, and our customers can reach us at any time if needed.
Technical Details
We provide a modular suite of software from video recording to real-time tracking, whether one tracks in single-view using a set-up with 1 camera or multi-view using hardware with multiple cameras.
Our whole system relies on video-based face tracking, so if you have videos with a face in it - either professional videos shot with head-mounted cameras or simple webcam feeds - you can assemble a pipeline with part or all of our tools.
Grabber software lets a user record and monitor shots from a workstation with accurate timecode and audio support.
Grabber also includes a real-time facial motion tracking feature that can be enabled or disabled depending upon the user’s needs and requirements.
Monitor lets the operator control remotely multiple actors on a motion capture set. It can receive triggers from body mocap systems, dispatch commands, use naming conventions, and synchronize everything.
Performer is our core software. It delivers motion tracking of an actor’s face from videos captured with Grabber software (or any other recording software), and solves for animation for the user’s target character with production-level quality.
Under the hood, Performer uses a machine-learning algorithm: from actor’s specific data provided by the user, Performer learns to extract motion for this specific individual with optimal quality, and transfer it to the target character in Maya, 3DS Max, Motion Builder or Cinema 4D with the highest fidelity.
The workflow first requires that the operator places predefined landmarks (annotations) on a set of relevant expressions for an actor. By doing so, he creates a tracking profile dedicated to the actor.
Second, the animator creates retargeting links: she/he matches the key expressions of the actor to the controllers of the 3D face in Maya, 3DS Max, Motion Builder, or Cinema.
Once this last step is achieved, the animator can process and check the animation sequence. The resulting animation data hardly ever need any cleaning if a good actor/character profile is achieved. If not, the animator can still refine the tracking profile by adding more key expressions or rework the retargeting links.
At this point, the job is almost over! Any new video sequence can be tracked and retargeted automatically.
If the animator wants to add a personal artistic intent, he can do so by adding anchor frames (kind of keyframes) on the timeline within Performer.
The same trained profiles can also be used for real-time tracking and solving, without any additional manual work. Our system is fully compatible with Unity and Unreal, for which we have developed a proprietary Livelink plug-in, but also Motion Builder, Maya, and many other engines.
Dynamixyz also features an alternative technology, Live Instant, for when quick fast turn-around is more important than animation fidelity, or when setting up tracking and retargeting profiles is not possible. Live Instant does not learn specifically from one actor as described above, but instead has been pre-trained on a database of millions of faces, and can deliver immediate animation results, in real-time without any setting or calibration. Such “generic” tracking and animation systems are very popular at the moment (iPhone’s ARKit, Google’s ARCore, etc.), and they are indeed very practical and easy to use. Yet, fidelity and accuracy are clearly lower than with the specific profile construction of Performer. The good news is that we can provide both alternatives, so depending on the project’s quality requirement, time, or budget, producers can choose one or the other.
Our software comes in 4 different packages: Live Instant and Live Pro for real-time, Performer for production, and Dxyz Factory for both production and real-time.
Hardware-wise, we have designed a whole range of head-mounted camera equipment that goes from a simple mount on which you can clip a GoPro camera to a multi-view Studio HMC.
Our mounts are light, comfortable, and fit any morphology. Our Studio HMCs are equipped with light-strobing LED rings designed to cope well with any ambient lighting conditions. Lately, they have been upgraded with a smart sync box that synchronizes camera shutters for Multiview tracking and receives timecode from any proprietary or external TC server. An LCD display on the mount gives the mocap operator a lot of information about the recording status and settings (FPS, TC, Exposure, Gain...).
Moving to the Technology
Live Instant is a good package to start off with for beginners. As its name suggests, face tracking is available instantly. The animator just needs to retarget the predefined poses onto its 3D rig and then animates his character in real-time.
We have a special operation going on at the moment so that animators can experience this solution for free during the #stayathome period. It is available by registering on the following web page.
Moving to a regular Performer has a learning curve, but it is also easy. We regularly host 15-year-old 10th-grade students who come for an observation period of 3 days. When they leave the company, they have learned the basics of the system, recorded themselves, tracked their own sequences, and retargeted them on an avatar.
Complete beginners to the system usually learn workflow within 2/3 days.
The most important thing to understand is that you do the job - creating a tracking profile for an actor and creating a retargeting profile for a character– only once for an actor/character pair. Then, you benefit from the system’s ability to automate from your initial input.
What you invest at the beginning will save you multiple times over in the end.
You do not need tons of frames annotated. 50 to 80 are usually enough depending on the variety of expressions you need, the targeted quality, and the complexity of the facial rig. Start with a minimum of 40. You can still refine the tracking profile by activating or deactivating frames if you find out that some poses are less or more relevant.
Parts of the tracked shots can also be ignored if performance was not great or occlusion happened, and you can always manually override results directly into the software by adding anchor frames.
We want our software to give animators maximum flexibility and act as a catalyst tool for high-quality animation for CG artists.
Free online video tutorials are available on Vimeo and YouTube. We like to get feedback from our users. They are very valuable to us as we know there’s always room for enhancements and advanced features.
About the Capturing Process
The tech does not require much. Although we recommend using our HMCs, you can use our software with any video-recording hardware. It even works with a regular webcam (or a camcorder, DLSR…).
Our software solutions are very versatile and can fit into any pipeline. The baseline level of delivered accuracy is quite high, making better use of the animator’s time and cutting production costs. That’s why AAA video games studios like it so much. Some of them are also heavily relying on our batch mode letting them process massive amounts of footage without supervision. A studio like Visual Concepts, for example, processes millions of frames of facial animation for its NBA2K games titles. This feature appears to be a true time-saver.
Another interesting use case relates to VFX companies. Compared to game studios, such companies are not looking to process high volumes, but rather to get accurate raw animation fast, enabling them to focus on animation refinement. There will always be manual animation polishing for high-profile shots in big VFX movies, and getting a clean animation draft as quickly as possible speeds up the process. Our software is a convincing tool for that. Before deciding to involve Performer software in the animation of Smart Hulk’s face in Avengers Endgame; Framestore first tested it with video recordings of Mark Ruffalo collected on YouTube. They were quite happy with the results they got and decided to move forward with Dynamixyz for the movie shots.
Courtesy of Marvel 2019
Courtesy of Marvel 2019
Our solutions are also easy to use for real-time purposes. We assisted the creative studio Big Company during the live performance capture of actress Anouk Hautbois, the french voice of Ladybug in Disney’s TV-show Miraculous. We captured Anouk’s range of motion and built tracking and retargeting profiles for the Ladybug 3D character. Everything was set up within a day. On D-Day, we simply had to push a button to trigger the real-time animation stream. Hardware-wise, the actress was equipped with a tethered Dynamixyz HMC and an XSens suit. The video was all processed with Grabber and resulting animation would stream live to Unity, and simultaneously to YouTube. Anouk could answer kids’ questions live on the YouTube chat through her character Ladybug. It was a very inspiring and exciting project!
Price and Future Plans
Our production software Performer starts at € 4,905 a year for studios and our entry-level Head-Mounted Camera at € 4,500. This can be purchased by contacting our sales team in France or through a local reseller listed on our website.
Our product roadmap is pretty packed right now. We have exciting new features that will come out this summer to reduce the time spent on manual work in our core software Performer and keep increasing its productivity.
Yet, there is a lot of other exciting stuff we are working on. The motion capture industry is evolving fast. High-profiling productions increasingly rely on it, but we also see motion capture becoming a much more democratic tool, fueling productions in studios of any size now. Even individuals now generate animated content with very few resources. The challenge we take at Dynamixyz is to provide the right tools for each of those use cases.
We hope our Live Instant technology will bring more people and studios to facial performance capture. For high-profile studios, we continue pushing the development of the scan-based system we showcased last year at Siggraph. In short, this system makes it possible to train the computer vision algorithm automatically from very accurate 3D scans of the actor. We are working at making it real-time. This system will deliver very high-quality animation on photo-realistic characters.
As technologies evolve for both hardware and software, we are seeing more and more algorithms embedded directly into portable hardware. We’re working on strategic partnerships to make that happen. Both our generic and specific trackers have already been embedded in Standard Deviation’s hardware so far, and data can be recorded and processed directly on the actor.
This is a breakthrough that will let operators track multiple actors at the same time regardless of video transmission systems limitations.
Last but not least, we are working at lowering down the retargeting processing time using deep learning techniques to enhance the results for real-time applications.
There are a lot of challenges, the technology is evolving pretty fast, we constantly need to rethink the solutions we’re designing. That is what makes this industry so exciting to work in!