Melinda Ozel Shared a Deep Breakdown on Being an Expression Scientist

Introduction

Hey! I'm Melinda Ozel. I specialize in the mechanics and behavior of facial expressions. There isn't an existing career for what I do. So for the past eleven years, I have been designing my own profession as an Expression Scientist.

I studied Biopsychology and Neuroscience at UC Davis. In a full-time capacity, I have worked at two tech companies, Meta and Afectiva.

Since I was at Meta for much longer than I was at Afectiva, my biggest project contributions come from my Meta days where I:

• Influenced the expression design of all smileys in Facebook's Emoji 2.0 (current emoji set);

• Advised artists on the animation and expression design of all Facebook reaction smileys (wow, angry, sad, and care);

• Contributed to foundational data initiatives and algorithm development for the face-tracking technology in Meta Quest Pro;
• Co-invented two patented technologies: 1) Systems and methods for improving animation of computer-generated avatars; 2) System and method for applying an expression to an avatar;
• Co-designed early renditions of Oculus Avatars (not the ones popularized by bad press in the summer of 2022).

Starting My Career as an Expression Scientist

I'd say I stumbled onto my career path through a series of related interests. I originally intended to be an English Literature major, but right before college, I came across a special edition of TIME magazine on the mind and the body. That one little magazine piqued my early obsession with brains and cognition, which led me to pursue Biopsychology and Neuroscience.

My original inspiration to study faces came from one of my psychology courses: Psychology of Emotion, taught by Dr. Wesley Moons. Dr. Moons's class introduced me to the world of facial behavior and FACS (Facial Action Coding System). At the time of taking his class, I was in a neuroendocrinology lab studying animal models for depression. It was thrilling to be in a wet lab and wear the stereotypical white lab coat, but animal models weren't for me. I was more of a human observer. When I found out that a validated system for observing facial expressions existed, I knew I wanted to become a FACS expert. 

After graduating, I hunted down the only FACS internship that existed back in 2012. It was held at the University of Zurich – the Zurich Interaction and Expression Laboratory (ZIEL). The first portion of the lab involved mostly self-study of the FACS manual, but there were peers and mentors available to help as needed. Once I finished studying the manual, I tested and passed the official "FACS Final Test" (also known as the FFT). Passing the FACS Final Test allowed me to become certified in FACS. Having a FACS certification indicates that you are qualified to conduct facial coding in academic research. It brings significant credibility to any studies using FACS-based data derived from facial coding. "Facial coding" is the act of using FACS to document which facial movements (action units, or AUs) a person is performing at a given time. Facial coding enables us to find patterns in human facial behavior. This type of research is how concepts like "basic emotions" were defined.

Post-FACS certification, the second portion of the internship involved facial coding for academic research. If you've ever watched any form of media glamorizing FACS (like the show Lie To Me), it should be noted that in real life, facial coding often requires over an hour to thoroughly analyze less than one minute of footage. It's a very tedious process.

Upon leaving ZIEL, I was eager to embark on my career journey. Unfortunately, there wasn't much demand for my new skills. For over three years, I struggled to find work relevant to FACS/facial expressions – or anything remotely related. Expertise in FACS/facial expressions was seen as (and may still be seen as) a bonus specialization to have on the side. Since there aren't roles for pure FACS experts, you typically need to also be something else like a user experience researcher or academic researcher. This reality made the early part of my career extremely discouraging. While I was grateful to find some contract facial coding work for a neuromarketing (market research that uses biometric data) company based in Minnesota, the projects were scarce and not nearly substantial enough to pay any bills. To supplement my income and to survive, I had to go into sales – commission-only sales. I was a literal door-to-door salesperson for a while. I would never recommend that kind of work! 

Being in sales was a less-than-ideal profession for me, but it helped to reframe my situation as informally collecting real-world data in semi-controlled settings. I did eventually wriggle my way into event-based sales where I developed a knack for assessing whom I could and could not close. If someone did a dimpler and/or lip presser, AKA AU14 or AU24, while I was pitching to them, I quickly learned they were not closeable; so I was able to allocate my energy toward potential buyers much more efficiently. Though my pitching and closing skills became quite instinctive, commission-only sales were exhausting. I reached a breaking point where I got so depressed and burnt out that I could barely get myself to go to work at all. Shortly after my breaking point, I was fortunate enough to receive a call from Oculus. They wanted someone with FACS experience to join full-time on contract with their VR face-tracking team. That call changed everything. I went from door-to-door sales to having a foot in the door with tech.

Some Words about Being an Expression Scientist

Given that I designed my own profession, it is no surprise to hear that my trade may be confusing. To my knowledge, I'm the only one doing what I'm doing – at least in this very specific manner. There are a ton of people with amazing knowledge of facial expressions in tangential fields. We have artists, psychologists, plastic surgeons, nonverbal behavior experts, etc. Perhaps you could view the work of Paul Ekman and Erika Rosenberg as close relatives, but I consider their contributions to be more on the psychology/behavior side, and mine to be more focused on art, anatomy, and technology. 

In terms of what this kind of work entails, on the client side, it involves:

• 1) Educating and training studios on facial anatomy and FACS;
• 2) Reviewing facial pipelines and advising on how to improve them;
• 3) Critiquing facial performance – holistically with the body as well (it's a working system after all);
• 4) Informing character design and expression design;
• 5) Helping strategize how to effectively generate and leverage facial data to develop tracking technology.

On my own side, my work heavily involves observation and research. I am a huge proponent of multidisciplinary research. Every field has a different perspective to offer, and the more fields you dip into, the more comprehensive your knowledge and understanding will be. I sample research from plastic surgery, primatology, dentistry, dermatology, zoology, and beyond.

The main challenges of this type of work are twofold. Challenge number 1 lies in the nature of the subject matter. Faces are vastly complex. The more I know, the less I feel I know. Our faces have so much variation that there are never easy answers. You just have to keep learning, observing, and expanding your mental model of how expressions work. Challenge number 2 is maintaining a steady income. While I have a very supportive community behind me and a solid roster of clients, I basically have to function as a social media marketer, web designer, and content creator to stay relevant and attract projects.

Experience of Working on Face-Tracking Tech at Meta & Affectiva

I touched on my face-tracking work at Meta a bit above when discussing my past project contributions. But to dive a bit deeper: I was part of the early VR face tracking team at Meta – specifically at Oculus. In the beginning, it was only me and three engineers.

Early on in the role, I was mostly working on defining landmarks – e.g. How does one define and label the boundaries of an eye? How many points do you need to annotate human lips? How can you define such parameters to be robust enough to remain consistent with different VR camera angles and prototypes?

My later work involved leading data pipelines.

• I developed our facial capture scripts and expression protocols;
• I created the documentation that allowed us to scale up our data labeling initiatives;
• I trained and oversaw third-party annotators at a mass scale.

I also worked very closely with engineering to ensure our algorithm development was on the right track. From working closely with engineers, I was able to co-invent two patents while at Meta (listed earlier).

In terms of what it is like to work at big-league companies, it was pretty amazing to work at a tech company during tech's peak. The people were friendly and welcoming. You had the opportunity to work on huge projects that could impact up to one billion people. The amenities were wild. There were vending machines that provided you with free devices whenever you wanted – keyboards, phone cases, chargers, batteries, name-brand headphones, etc. There were all kinds of high-quality meals from different chefs at different restaurants. Every floor had a micro-kitchen adorned with snacks and espresso machines. We had an arcade, a dessert shop, a rooftop smoothie bar, and probably a bunch of other things I forgot about and/or took for granted. Everyone described it as a workplace Disneyland. They weren't wrong.

What Services I Provide as an Expression Scientist

The services I provide are varied. If you are really specialized, at some point, you get so niche that your work goes from narrow to broad. You'd think that hyperfocus would be limiting, but if you can brand your work well, you can make it expansive. To actually answer the question – in simple terms, I help companies build naturalistic and expressive characters in any level of stylization. This work could involve anything from critiquing FACS shapes to helping strategize rig development to overseeing facial performance.

How I work with animators depends on the project. For example, when working on photoreal nonhuman primates – a CG capuchin (Gigi & Nate, film) and a CG chimpanzee (Jane, Apple TV show) – I had been heavily involved in the modeling and rigging process; so I taught animators how and why the facial rigs were built the way they were built and how to use the rigs to achieve certain expressions. I also did deep dives into primatology and primate behavior to teach animators what kinds of facial expressions nonhuman primates make in various contexts and what those expressions mean.

RE: Lectures & Training – Most of my lectures involve FACS training for studios. These lectures can range from a speed-talking two-hour overview to a multi-month series. They are typically attended by modelers, riggers, and animators. My lectures are general enough that all types of artists can benefit from them but specific enough to inspire actionable improvements.

Companies looking for general training opt for the shorter, two-hour session: my FACS Cram Session. However, companies more dedicated to naturalism and anatomical diversity opt for my deeper dives. Epic Games was the first client to take on my anatomical variation lecture series. That was a really fun project because there is SO much to cover beyond FACS. Whenever I have an excuse to get more into the weeds, I love to take on that challenge.

RE: Project Consulting – Outside of lectures, I essentially do anything that involves making faces look better. Sometimes, that entails helping studios strategize their facial pipelines and improve their existing expression sets. Other times, that means actively reviewing facial performance, building facial pose decks for data collection, participating in facial data captures myself, helping pinpoint what looks "off" and how to fix it, etc. Now that the landscape is changing, I have found myself dipping into different types of AI projects. I recently worked on improving deepfake technology to de-age a celebrity for an upcoming film. That was very interesting because I ended up becoming a "black box interpreter" for the AI. It was exciting to figure out why the AI was making certain decisions and how to redirect those decisions through data and other forms of manipulation to eliminate as much uncanniness as possible.

My Experience with Giants Like Epic Games, Blizzard, EA, & More

I worked for all of these companies as a freelancer, and it was awesome. The only challenges were that the engagements were too short, and I didn't get to stay and hang out with them longer! Well, that and IP issues. Big companies have a lot of restrictions on what they can actually show me, which can be a slight obstacle when I'm trying to help them improve their systems.

I got approval from EA and Blizzard to detail some of our collaborations. I'll list out some bullets below.

With EA I:    

• Worked directly with one of EA's advanced research groups;
• Consulted on how to increase fidelity and efficiency on next-generation facial capture pipelines;
• Provided multiple ROM scenarios for different capture needs, durations, and environments.

With Blizzard I:

• Reviewed their facial pipeline and helped them enhance their facial performance;
• Delivered live FACS-based and facial anatomy lectures;
• Analyzed and critiqued footage from current and past titles;
• Developed bespoke educational content integrated with analyses and critiques for current and future teams to reference/train with;
• Advised on long-term standardization for best practices and principles around facial rigging and facial animation.

To learn more about my work with Epic, I have already detailed this a few times on my LinkedIn.

How to Start with Expression Science

I can't tell someone how to forge the path I custom-made based on my own personal and hyper-specific interests. I can, however, offer some general advice regarding career development. If something deeply interests you but there isn't an existing job for it: Pave your own path.

With the rapid advancements we will continue to make in technology, there is always room for new types of work and roles. You do not have to put yourself in a box. It is important to note that rebelling against being put in a box is not easy. You will have to convince companies they need your skills. You will likely have to wear many hats, take odd jobs, learn how to market your work, build a network, and become your own content creator/web designer/social media marketer. All of these things will take time.

The benefit of fitting into a box for existing roles is that you don't have to take all of those extra steps to prove yourself. You can take the traditional resume-to-job route. If that's not for you, be prepared to put in extra work and have patience. It takes a while to see results. Even after building a following and having the privilege of working with big-name companies, I still have to fight for career stability and longevity. I am so grateful for everyone who has supported me on this bumpy road. Without the art community, I would have likely had to do what I had been tempted to do many times: Give up. So, thank you. Thank you all so much.

With all that said, if you specifically want to do something similar to what I do, start with the FACS Manual. Gain credibility through the FACS Final Test. Spend years observing faces and never stop. You will also need to study facial anatomy and behavior!

Melinda Ozel, Expression Scientist

Interview conducted by Theodore McKenzie