Confessions of an Unreal Engine 4 Engineering Firefighter

I have been working professionally with Unreal Engine for over 9 years. During this time, I have acquired many specializations and have fulfilled various roles in game development: from grunt engineering to managing large engineering teams for games or even consulting investors for game teams.

Lately, I have been working for myself, but occasionally I will offer an emergency white-label “firefighting” service to my word-of-mouth clients. It is hard to describe what this service entails exactly, but it is a lot like having an emergency plumber. You really do not want to be in a situation that requires calling for one.

If a game company in Los Angeles is having an Unreal Engine 4 problem that no one can solve, I usually end up getting that call. I am writing this article to explain why I get that call, how one would prevent the need for that call, and what I normally do after getting that call.

Most game development problems are well understood by those “in the trenches”, but those problems fly right over the heads of those on the management and executive ladder. It also seems like the people in those trenches are the ones who read articles like this one, instead of the others who need to.

This article is most likely irrelevant to you if you belong to a small fully indie team. Most of the evidence provided is anecdotal, thus validity and mileage may vary. Due to the nature of my work, I am often uncredited; the mere fact that I am involved with fixing a team’s mistakes is kept confidential. This is exactly why most of my clients are referred through word-of-mouth, and this is also why there is no public proof that I am even qualified to write about this subject. I am writing this regardless of whether I can be believed, as the people who truly need to know this information will not read about it until it is too late… which is in a nutshell exactly the problem I’m writing about.

Stop Burnout, Start Growth

In the Los Angeles area, there is a severe and unmet demand for high-level Unreal Engine 4 specialized game engineers. Incredibly often, I have seen teams hire one or multiple entry/mid-level engineers, and then immediately try to extract senior/higher-level work out of them, without offering them the room to grow or fail. This is due to teams generally agreeing to deliver a product before having the means to build a said product.

I believe that this is causing a rapid turnover and extreme burnout: it is a negative feedback loop that crushes the potential for high-level local talent, thus creating even more ridiculous demand. If you find yourself in the Los Angeles area, I recommend attending any local game dev meetup, where plenty of alcohol can be found, to seek out the engineers; they will surely have plenty of incredible and insightful information to share about this subject, for the cost of a drink.

Overtaxing your engineering effort is a surefire way to end up with hard-to-maintain work. Unless you are developing a throwaway prototype, hard-to-maintain work is one of the fastest ways to cause engineering fatigue, short from cultural or environmental problems. It does not matter if your office has a Nintendo 64 connected to a 300-inch projector if your engineers have to spend their working time making and eating blueprint spaghetti for breakfast, lunch, and dinner.

The biggest creator of hard-to-maintain work stems from having someone design a gameplay system, except with underdeveloped engineering skills on a gameplay system scale, resulting in tightly coupled systems that are held together by rubber bands and chewing gum. Games require many systems to work together; without having someone with experience in systems design or with a game engine’s whole ecosystem, those connections suffer. This leads to situations where the person who has created the system or connection is the only one qualified to maintain or make changes to it; this stacks a permanent responsibility onto this engineer and prevents them from being freed up for other tasks. Eventually, this engineer may carry more responsibilities than they can handle, and development suffers from this person being fatigued, or worse, leaving altogether. Neither the engineer nor team can grow because of this, as new features or changes get increasingly more resource-demanding to implement.

This problem is often discovered far too late due to not having a higher-level engineer be able to identify it. Instead, this problem is discovered at a critical point where what you have and what you need become connected, by a narrow and long wooden bridge that just happens to be on fire, requiring an engineering firefighter to save the day.

If you are forced to hire engineers at a lower skill level than required, there are still ways to prevent this problem. Instead of forcing these engineers to attempt to create something above their skill-level, break the required project down into smaller systems, possibly cutting some features in the process. Bring on an extra engineer or two to split the work more evenly across your engineering team. Have your engineering team set up isolated testing of their systems, and isolated testing of specific combined systems. Not only will this help the QA team, but doing so allows every engineer to identify where a problem is, even if it belongs in a system they did not create. Creating isolated tests of specific combined systems will help your engineers grow their system design skills, their trust in their fellow engineers, and their ability to work on codebases as a team rather than as an individual. Breaking things into smaller systems, and having engineers work with each other. Creating these combo tests also allows multiple engineers to be exposed to a system, reducing the reliance on a single engineer to maintain a specific system. This frees up engineers from being locked down to certain responsibilities, allowing them to grow and produce more engineering effort. If an engineer is lost, your means of engineering should not be lost either.

A contrived example of this would be having one lower-skilled engineer work on a system that allows players to pick up keys to open doors. Another lower-skilled engineer would be working on some sort of loot or item-dropping system, invoked when NPCs die. Your most skilled or weapon-specialized engineer would be working on allowing the player to hold and shoot a weapon. All these mechanics can be tested in isolation as well as in combined QA tests, i.e. shoot this guy to pick up a key to open that door. These systems should be able to work together without issue, and without entanglement.

I have picked this example because I have seen what happens when an engineer, without system design experience, tie all three of those systems together; a tangled mess and the potential issue of every door requiring an NPC to die. If a door requires a key, and all keys are dropped from dead NPCs, and the only way to kill a dead NPC is to fire a weapon, you’ve essentially locked your game design to needing to shoot a weapon to indirectly open a door. If this sounds ridiculous, it is. But these types of ridiculous issues and dependencies happen all the time. Depending on how the concept of a key was implemented, the simple act of allowing a level designer to place a key that can be picked up could be an incredibly expensive feature to implement. Or worse, it could require a separate system, duplicating the amount of engineering work related to the game concept of a key. This is especially true when the problem is scaled up, with 10-20+ systems working together, and the only person who knows their way around them is no longer available for some arbitrary reason. Imagine having a game team with only 3 days left for delivery, and they no longer have their engineer responsible for those systems, and your team of lower skilled engineers are incapable of learning how to maintain and change these systems. What do you do? Who do you call?

Not Having an Engineer Is a Problem

This should be an obvious problem but a lot of my client work stems from this, sometimes connected to the previous one. An executive or producer might know that they need a senior engineer or technical lead, but they end up giving up when they cannot find one and fall into the trap of being content with their current engineering effort. “We can’t find someone but maybe our team can handle it.” If the engineering team is given room to grow so that someone may potentially step into this role, chances of success are greater as previously described, but this generally does not happen. Sometimes management might skip this search for a higher-level engineer altogether and go straight to settling for less, or worse, decide to figure out the issue later.

In either case, we end up in the situation of a game development team developing without a senior or lead engineer. Without this role to prevent the overtaxing of lower-skilled engineers, as well as without all the foresight to prevent engineering issues in general, the chances of discovering a critical issue far too late in development increase dramatically. These issues range from the most trivial issues to huge overarching design problems that end up lighting an entire project on fire if they are not dealt with.

One recent example of a trivial issue cascading into a costly production nightmare comes from a client of mine that suffered from a source control server misconfiguration. The problem was essentially that of temporary and local cache files, that are never meant to be shared with a team, but that were being tracked and distributed needlessly. Any high-level engineer upon seeing this would immediately point out that these files should remain local, and never ever be sent to source control. They would remove these files for all your standard reasons of not bloating a source control server. The act of doing this would have prevented a much more nefarious issue from popping up.

It turned out that if the game engine detected these cache files being present, the engine would not generate the data it needed to play certain forms of audio correctly, as it would assume that it had already generated this data, because why else would it have this cache? This caused the audio to only work on the machines that were authoring audio. This issue was compounded by the fact that the audio was being created by a sub-contractor and not in-house. When telling the audio guys to fix their audio, but the audio guys report that everything was working as expected, this could lead to cascading communication errors and needless company drama of everyone blaming everyone else, and making the workplace feel hostile; from potential lawsuits about the sub-contractor not delivering, to everyone putting the blame on the poor engineers who did not know any better, and did not have the experience to even begin to diagnose the issue. When I got the call to fix this dilemma literally a couple of weeks before release, the simple act of cleaning up the source control server, and purging these unneeded cache files from source control, fixed this audio issue that was plaguing them for months and had cost the company tons of money and time.

This is just one example of literally thousands that could occur without a senior engineer or technical lead. Literally, the cost of paying this role’s paycheck would have been a lesser expense than having this on-going problem to be solved weeks before release, even if the only thing this engineer did was say “hey I don’t think those files belong on source control.” Even in the case where this slipped by this engineer, this engineer would at least have had the experience to fix this without requiring an emergency consultation. If this issue was not solved, a lawsuit may have been wrongly filed against the sub-contractor, which would had led to astronomical costs over a trivial issue. Unfortunately, this situation occurs more often than one would think.

Learn from Your Mistakes

Imagine a company that works with budgets of a million or higher, repeating the above issues multiple times. Not only does this happen, it never seems to stop happening. This was a big motivator for me to write this article because it saddens me to see this occur. In addition to the incredible wasteful spending, it also often meant the wrongful firing of engineers who were not only overworked but were achieving work higher than their skill level. If those engineers were given room to grow instead of being let go due to what ultimately came down to management failure, those engineers could have become their own powerhouses, whose skills would allow an engineering effort to survive even in the face of management failure.

This is not to say that underperforming engineers should be kept on staff. What I am trying to make clear is that if your company is to deliver a product that requires an engineering effort, you should also try to distinguish between an underperforming engineer, and a handicapped engineer, who is overburdened with responsibility in relation to their skill level. Hopefully understanding the two problems mentioned previously will help you distinguish this.

High-Level Engineering Is More Than Just Code

A high-level engineer will often look outside the code to diagnose issues with it. This is especially true when many moving parts are involved in the development of a project. Knowing the background of all parties involved can be especially illuminating.

Over time you can start to see patterns with how certain companies work, and if a company says sub-contractor X is causing problem Y, and if you have previous experience with X’s work, you’re one step ahead of solving problem Y. While this information can be invaluable, unfortunately, unless you have been doing this type of work for a while, you won’t have a prior catalog of information to sift through. This can sometimes be solved by the simple act of buying another engineer a drink, as this article originally pointed out. Engineers usually will not break NDA but get gossip about how a company operates is much easier than one might think. I cannot count the number of times I have solved large-scale problems by simply finding out from an ex-engineer, or a friend of a friend of a friend, that company X says they do one thing, but in-fact do another. It is hard to write about these types of problems though, as they tend to involve very specific people at very specific places with very specific problems; writing about them would go against the discreet aspect of my work. Sometimes background information can be used to solve issues ranging from who has the correct animation files to what happened to their millions of dollars.

Here is one example that might be hard to read with names, numbers, and minor specifics changed.

Company Moose was hired by Publisher Squirrel to make Game with a budget of $1,000,000 over a 2-year time span. Publisher Squirrel stipulated that Company Moose must use Contractor Rabbit to do their concept art, and Contractor Bird to do their animation work. Both Contractor Rabbit and Contractor Bird’s fees were to be paid from this $1,000,000 budget. Company Moose accepted these terms, even though they were advised not to do so by Engineering Consultant Bob, who said that realistically they would need $2,000,000 to develop Game. Company Moose ended up taking 3 years and spending a budget of $2,500,000 to achieve an underwhelming and partially incomplete Game. I was brought in at the end of these 3 years by Company Moose, to see if there was anything that could be done to fully develop Game with a minimal budget, and to figure out what went wrong. After digging up information about all parties involved, it was pretty trivial to prove that Publisher Squirrel had Board Member Joe, who had a hidden stake in both Contractor Rabbit and Contractor Bird, and who had maliciously told both Contractor Rabbit and Contractor Bird specs that required far less work than what was actually required, causing them to underdeliver, while knowing that the engineering team at Company Moose was not skilled enough to realize that the work coming in was not up to par. This information was gathered by asking the grunt engineers what they thought was going on, and by asking Engineering Consultant Bob for help at a local game company’s open bar event. Because Company Moose signed off on the work that was given to them by Contractor Rabbit and Contractor Bird, legally it was Company Moose’s fault that Company Moose ended up needing to pay overages to get more work done from Contractor Rabbit and Contractor Bird, inflating profits for Board Member Joe. This was a potential red flag that was correctly flagged by Engineering Consultant Bob, but Bob was merely a consultant and Company Moose did not have an engineering lead to confirm this with. If Company Moose had a high-level engineer, they may have backed out of this deal, or at least been able to identify the sub-par work coming in from Contractor Rabbit and Contractor Bird, and would have been able to dispute the issue with Publisher Squirrel based on Board Member Joe’s forced stipulations. Instead, Company Moose ultimately ended up being royally fucked by Publisher Squirrel, and Company Moose now no longer exists, whereas Publisher Squirrel is continuing their bad practices to this day, except Publisher Squirrel is now Publisher Trash Panda.

News outlets fully agreed and reported Company Moose was solely at fault with zero details regarding Board Member Joe, Contractor Rabbit, or Contractor Bird. The whole truth regarding these situations never comes out and is usually skewed heavily towards whoever is responsible for the engineering effort, regardless of whether other malicious actors are at play.

By the Way, All Your Subordinates Are Lying to You (Because You Don’t Listen)

One very interesting thing I had found, when I came into an office as an outside objective evaluator and you had left me to do my job, was that approximately 2 nanoseconds later your employees were already telling me everything that was wrong about you, your company, and your development methodologies. It was not a matter of having a trusting face or even promising them to make things better, it was about your employees who cared about the growth of the company, and more importantly about their own growth within a company despite feeling unheard and ignored. I just happened to be a new face that wouldn’t judge them for what they would say, and even if I did, often they are so tired of the problem being present that they would even just stop caring about any potential consequences of speaking out against their employer.

Usually an outstanding problem or issue is already known by your staff, and often they have tried to make you aware of this problem, but usually, it gets prioritized into nothingness or filtered into white noise by your chain of command where no action gets done. Worst case is when someone brings this up directly to the most senior decision-maker, and it gets either immediately rejected or the intent to ignore is made clear.

Allowing reported issues to result in zero action is the quickest way to convert your employees, especially your engineers, from enthusiastic company-first workers into “clock in, clock out, it’s not my problem” workers. Most employees, again especially engineers, want to improve the company and themselves… until you prove that their thoughts do not matter.

One example of this was a company’s project that was taking much longer to develop than it should, and I was called to either speed up development through engineering, or to figure out what the issue was. Immediately after arriving on-site, and before I had been greeted by the person responsible for hiring me, several engineers were telling me to run away as fast as I could because the producer who had complete authority over engineering has no engineering knowledge, and kept mixing up and reassigning tickets to the wrong engineers; network engineers getting gameplay fixes, gameplay engineers getting rendering fixes, etc.. Because of these mix-ups, frequent meetings were held to fix these mix-ups, and if engineers hate one thing, it is frequent meetings. Every one of those engineers had claimed to have brought up this issue with the producer and upper management, and they had even shown me emails of this communication, without any prior checking or confirmation that I had to be given access to such information. They were so exhausted from the situation that they just simply started doing what their tickets said. One of the network engineers spent 3 months learning and working on animation graphs because of this. In this case, engineers found a way to improve themselves but unfortunately without improving the company. Because I was able to discover this before even taking the company tour, I was able to lay this all out in front of upper management immediately. They immediately asked who was giving up this information and wanted to crack down on this perceived insubordination, rather than trying to address the issue. I was then dismissed for not specifying names, and my services were no longer required. A month later, their engineering team went from 7 engineers to 2 as they were abandoning ship. A month afterward, I got a call saying they no longer had an engineering team and asked my rates to finish the engineering effort. Their offer was nowhere near the needed budget to hit the required engineering milestones. They ended up hiring a whole new team of lesser skilled engineers, and the process repeated itself.

Another company in a similar situation reached out to me due to my work with one of their close friends. Their response, however, was to put everyone in a room, allow everyone to re-hash their issues in the workplace, and immediately act. This company has now tripled in size 3 years later and are working on some incredible stuff.

As a manager, you can tell if an engineer is happy with their work if they are comfortable with giving higher-ups constructive feedback about issues at hand directly. As an engineer, you can tell if a manager is actively working for you if you see at least some action performed as a response to your feedback. When both things happen, management might see an increase in reported problems during development, but they’ll also see an increase in resolutions, instead of seeing fewer but critical issues that go unresolved. Burndown charts are a great tool for identifying which camp you are in. Assuming you are using issue tracking software correctly if you are seeing erratic spikes and inconsistent numbers week-to-week regarding your number of open problems to a number of problems solved, your team is not working at their full capacity. There is a good chance that there are underlying problems affecting your team. Sit with them and actually listen to them, but more importantly, take action. In many cases, even the wrong action for the right reason is better than no action, if you’re willing to admit a mistake. If your burn trajectory has a consistent slope (i.e. a straight line), your team is most likely operating as they should, and I would have no reason to be on-site.

All this being said, just because you have an outspoken engineer who shouts about development issues every day does not mean your team is healthy. If only one engineer on a team is causing problems, and you have no lead present, usually one of two situations is occurring. That person may actually deserve to be your lead and his/her fellow engineers are trusting them to make the changes needed, or that engineer is just an asshole. Sitting down with your engineering team, and treating them with respect, will quickly let you know which situation you are in. Treating them with respect and valuing your employees’ feedback is usually the best way to always identify the situation you are in.

If I get the call to come in and fix engineering issues, I immediately try to seek out the people who are clearly disgruntled or have already confirmed that they are finishing a 30-day or 2-week notice due to employment elsewhere. If I’m able to identify that they are correctly disgruntled, and simply not causing havoc, I’m going to feed you your employees’ own knowledge. And I’m likely going to charge more than it would have cost for you to simply listen to your employees.

When you do not listen to this relaying of information, and you are still set on hiring outside forces to solve your problem, I will do whatever engineering is required and absolutely shock you in terms of how productive one engineer could be. In reality, this is not because I’m a super genius high-level engineer, it is because I am leveraging your team’s engineering against you because you are too fucking thick to see otherwise. This goes back to my writing of “Learn from Your Mistakes.” I will give you every tool possible, and perform to the best of my ability, to help you prevent making the same mistakes. If you are not willing to learn, I am more than willing to charge extra the next time you call me.

It is important to note here that I am not trying to take advantage of your situation. Charging more is a direct response to being tired of lessons not being learned. Conversely, I do in fact lower my rates and increase my availability for companies that have proven to learn from their mistakes and adapt. This is because while they still may have a problem or are facing a situation caused by outside forces. Working with those who understand growth both as an employee and as a company simply feels good, and that is exactly what I want to do. I get to work on some very cool and fascinating projects that I would want to work on regardless of money because of this, and it is this environment that also keeps your engineers happy and not wanting to flee elsewhere, even in the face of competitive offers from other companies.

Don’t Become a GDC War Story

The story above? That is now a War Story. Most experienced game developers have War Stories. You do not want to be one. Trust me, if your company reaches a certain level of hell, everyone is going to know about it. Everyone. And if they don’t, they will soon.

Some of the following points are redundant because of how often they occur.

War Stories include but are not limited to:

• Incredible amounts of harassment of any kind
• HR who openly is on “The Company’s Side”
• Frequent unpaid art tests being used in production deliveries
• Doing business deals at strip clubs
• Doing job interviews at strip clubs
• Hosting employee parties at strip clubs
• Really anything company official has done at a strip club when a company isn’t stripclub-oriented
• Doing business deals while high on drugs
• Openly doing cocaine in the middle of your office
• Firing of an employee because you have a crush on their significant other and somehow this seems like a good idea to you
• The hiring of an employee because you have a crush on their significant other and somehow this seems like a good idea to you
• Literally, anything to do based on the reasoning of wanting to manipulate a situation with anyone’s significant other
• Allowing Person X to put their toenail clippings in Person Y’s desk almost weekly to the opposition of Person Y because Person X suspects Person Y once stole Person X’s Pepsi
• Being late on a payroll for weeks consistently
• Purposefully withholding employee mail
• Allowing employees to purposely go through other’s mail
• Having your company share a bathroom with another company who has clearly no respect for what a bathroom is and continuing to do nothing about it
• Verbally insulting staff who are clearly overworked and are suffering mental issues because of being overworked and consequently being verbally abused
• Punching employees, business partners, etc. for really any reason
• Creating a hierarchy structure based on who plays golf the best
• Just being an asshole and allowing asshole-like things to happen underneath you
• Not being able to afford giving employees access to drinking water but to continue to hire people
• Creating an atmosphere where two opposing sides of an issue are able to exist without any compromise from either side

Maybe we can start a #gamedevwarstory campaign or something.

Don’t Be A Victim Of Corporate Kool-aid or Cult Culture

If you’re a company with 50 employees, you should want employees who are capable of self-thinking and want to grow as an individual and to contribute to company growth, as discussed earlier. What you do not want is employees who are blindly following you down a potentially self-destructive path because they believe anything the company does is what it should do.

At very large companies, employees can sometimes get into this mindset and never escape. It is known that to work at some companies, you must succumb to their corporate kool-aid to survive. Hiring an employee to work on your game, having them doing unpaid overtime, and having that employee want to actively play your game during their lunch and off-hours can be incredibly unhealthy for everyone. It’s very easy for people and companies to stunt their own growth when all they see is their own product or idea.

If you’re potentially hiring from one of these large kool-aid drinking companies, it’s important to vet the candidates for their ability to self-think. What might be good for your company might not be what’s good for Company X, and if the candidate is unable to adapt, you might be setting yourself up for a potential war story?

As an employee, it is important to know what is good for the company and to know what is artificial flavoring. You should be actively agreeing with the way a company is moving forward, instead of passively letting things deteriorate. You might wake up one day with a hatred no one can soothe.

Moving Up As An Engineer / Growing Your Engineers

If you’re reading this as an engineer (but this may apply to many fields), or as someone looking to grow an engineer, the most important thing you can do is identify current issues and weaknesses. If there is only one lesson I can impact on the readers of this article, its that if you can identify and solve your problems as a whole instead of an individual, you’re going to be okay.

As an employer, try to coax out your engineer’s weaknesses without being overly blunt about it. Randomly bringing an engineer into an office room and saying “what are your weaknesses” is sometimes far too confrontational. Ease into the subject. Try to invoke some long and meaningful conversation. As an employer, lead, producer, etc., this is one of the most critical people skills you need to have to really make the most out of your team.

For those engineers reading this, please take note of the following. I’ve found that my opinions on this matter are somewhat controversial and many will argue differing points, but for those who are looking everywhere for something to reach out for in terms of knowing your place as an engineer, I believe this is a very good starting point. Only personal experience and war stories will show you the truth here.

As an engineer, you will always have a ‘weakest area’. What separates entry, mid, and senior level engineers are usually the types of weaknesses an engineer has, as opposed to their strengths. Exceptions to this always occur, but it is a good general rule.

For example, you know your way around C++ and Unreal Engine 4 and are fantastic at exposing gameplay systems to designers, but you lack the ability to communicate to your designers about what you have exposed. This screams mid-level, as you can be trusted to get the work done but the team will be relying on others to make sense of the work. If you need the support of people above you more than the people to your side to organize meetings and convey information, you’re mid-level.

If you are a guru at gameplay mechanics but this most recent mechanic you’re working on requires some fancy form of rendering or visual effect you’re not sure how to pull off, and you need help from those sitting aside you as opposed to above you, and you’re able to communicate this need and get the job done without the need of about a hundred cross-department meetings, there is a good chance you’re qualified to be a senior-level engineer.

If you constantly find yourself arguing with those at your side about implementation details but know when to compromise and are able to do so in a way that boosts morale instead of degrading it, there is a good chance you are a mid-level engineer transitioning to either senior-level or becoming a lead but are simply lacking the skill experience to see it through. Skill experience here would allow you to speak with more authority and to present implementation ideas that are more sound overall, requiring less debate assuming that you are correct in your implementation. Skill experience here also allows you to quickly back down and admit failure when you are indeed wrong about implementation details.

Becoming a senior-level engineer is about being able to recognize what is right and being able to deliver on it. Staying a senior-level is about honing your focus, staying disciplined, and not repeating mistakes. Making mistakes does not make you a lesser engineer, not learning from your mistakes does.

If you find yourself being able to work out problems as team incredibly well but feel like you don’t have the skill to be the one who frequently comes up with the “last piece of the puzzle” or the “critical problem-solving solution”, you might not be heading toward being a senior-level engineer but instead you might be better off as a lead engineer.

It is my belief that lead engineers should not be mistaken for senior engineers and vice versa. Lead engineers should be skilled, and they should be above seniors in structural authority, but that does not mean that they should be automatically paid more, valued more, or trusted more. The role of a lead engineer is to lead. The role of a senior engineer is to solve the tough problems. Ideally, your lead engineer is also a senior engineer, but this is not always the case. I could go on and on about what its like to be a lead versus a senior engineer, but this is not the article for that. It’s up to your producer to figure this out.

Of course, you might find yourself in a situation that is not listed here, as I’m only briefly covering what an engineer’s weaknesses may be, however, try to categorize your weakness in terms of who’s support you need to overcome it and how much of that support is required.

If you are the guy asking for help from people below, to your side, and above you, you’re probably an entry-level engineer. If you frequently need support from those above you (excluding C-level and executive support, that my friends are a whole new level of hell), there’s a good chance you are a mid-level engineer. If you find yourself supporting and being supported by those at your side, you are likely a senior or lead level engineer.

If you’re reading this as an employer or producer, this is how you should be classifying your team’s weaknesses and how you should look at distributing team effort to overcome them. If you find yourself in a situation where your top-level engineers are being strapped for time because their too busy helping the people below them, you need more senior level engineers. If you find that productivity is good and that you are underutilizing your senior level engineers, the answer is to bring on mid-level engineers. The seniors will allow the mid-levels to grow, which in turn allow the seniors to grow as well either sideways into lead roles or upwards into executive/principle/board member roles.

If you’re unable to classify your engineering weaknesses as a producer or employer, try to get help from an engineer who can and have them work with you to grow your own weakness identifying skillset. If no one is able to identify any weaknesses anywhere at all, you might want to leave and find growth someplace else.

Dumb Things That Bad Engineers Or Companies Without Engineers Do In Production That Hurt Themselves

When it comes to the pure engineering aspect of being an engineer, sometimes being an effective engineer is simply the enforcement of seemingly obvious things. If I am called to figure out a pure engineering problem, 95% of my time is usually spent identifying the problem, 4% involves fixing it, and 1% is about telling everyone that the problem has been fixed.

Any high-level engineer specialized with Unreal Engine 4 should know how to use the game thread profiler and the GPU profiler. They should be able to make blueprints and/or write C++ without making spaghetti. They should also be able to grasp what is happening across the entire engine across all systems at least a basic level. A manager of any role should be able to identify that an engineer is capable of these things. Getting to this point could be a whole article on its own, so instead, I’m going to list some of the most ridiculously common problems that I face every time I am hired for an “optimization problem” for a project. Hopefully, by reading through these issues, you might be able to understand some of the thought processes by identifying marks that lead to costly hard-to-maintain engineering, or to a clean and efficient engineering effort.

If you are an entry or mid-level engineer looking to move up, hopefully, this list will give you the confidence in knowing that a big part of taking the next step and moving up, is simply having the experience to point out ridiculously simple issues and communicating them effectively. This is not an exhaustive list by any means. These are just the most frequent and recent issues I solve for clients reaching out to me, some of which are painfully obvious.

• Use Source Control: Do you not use source control? Are you unaware of what source control is? If either of these is yes, you are on a path of pain. Have you ever heard of a game company losing their master copy of their development tree, losing months of work because someone spilled water on a running hard drive? Trust me, it happens.
• Smaller Commits, Better Commit Logging: Submitting a 2GB 2000-file change to Perforce with the description “did some work” is terrible. This does not help at all if we ever need to track down a problem you probably caused when you last submitted 2000 files.
• Make Builds and Test Them: Imagine a game has a 2-year development deadline. Imagine the first playtest happening 1 year and 9 months into development. This happens. Make regular builds of your game and playtest them, even starting from week one. If you are a serious game development company, you probably should have an automated build pipeline of some sort. Pro-tip: Automation is cheaper and more effective than human labor.
• Test Your Builds on The Target Hardware: This happens frequently with VR or mobile development houses that I am called to help resolve issues for. Imagine having 1 year to develop a VR title and you absolutely know your minimum spec is a Nvidia 960 GPU, then performing your first test on a 960 after 11 months of development and suddenly finding out that you’re nowhere near your performance budget. This happens at least once every 3 months in the Los Angeles area.
• Stop Making Spaghetti: I can’t emphasize this enough. If you’re working with Blueprints, stop the spaghetti. You’re hurting yourself for no reason.
• Give Your Engineers Time to Spruce Up A Rushed System: If you are rushing your engineers to complete a proof of concept or prototype, but then want to use that mechanic in production, you should have your engineers upgrade/remake the system to be more production ready. Quickly slamming stuff together is how you get project spaghetti.
• Try to Follow Some Form of Style Guide: You do not have to be perfect, but your project should be consistent across its entire development. Not doing so makes new hires so much costlier as now they have to learn your mess. Don’t have a style guide? Maybe give mine a go.
• Networking Is Not an After-Thought: If you know you are making a multiplayer game, make it support multiplayer from the start. You cannot simply just ‘add multiplayer’.
• Actually, Run The Profilers: If I show up, open a profiler, and immediately find you have a blueprint issue costing 100ms when it should take 1ms, I’m probably going home and charging you for the full day.
• Treat Warnings as Errors, Treat Errors as Sins: “Oh, that warning has been there for months, you can ignore it.” No. It is warning you about something for a reason. Ignoring warnings cause cascading problems. If you are ignoring errors, you are shooting yourself in the foot. They are errors for a reason.
• Don’t Ignore Your Lead Engineer if You Have One: If your lead engineer says that you need to spend $2,000 on SSDs or buy X license for Y, and your answer is immediately no because you believe your engineer is just trying to waste your budget, you either need to learn how to trust your engineer or find a new engineer. I have seen a company waste 70 man-hours a week simply because they had their workers on slow hard drives. Giving everyone faster SSDs would have cost the company ~60 man-hours in the budget. They let this issue go on for over 4 months, burning over a thousand hours of man hours because they did not want to pay for an additional sixty.
• It Works on My Machine: If you encounter a problem that ‘works on my machine but not yours’, do a file diff immediately between the two systems. 90% of the time the diff will show the issue. If the diff reports that both are identical, the issue may actually be system related.
• No, It Is Probably Not a Compiler Bug: The compiler is much smarter than you. If you cannot compile, always remember it is most likely you, not the compiler.
• 4k Textures Aren’t Always Needed: You really do not need 4k textures on every single asset unless you have a very specific need to. Consider texel density while mesh authoring.
• 1000 Dynamic Shadowing Lights Are Not Good: Stop flooding your project with dynamic lights. Yes, it looks fantastic at 3FPS, but unless you are doing some form of offline rendering, your game is not really a game at 3FPS.
• You Don’t Need 500 Pickupable Objects: No, the lag is not worth being able to pick up a single crayon from a box of 256 crayons. Unless your game is about crayons.
• Learn PBR Content Authoring: You’re making realistic assets using a 10-year-old Film asset modeling and texturing techniques? Invest in some PBR training if you must. You really do not need a 4k specular map for every asset.
• Your Rocks Don’t Need 12 Material Slots: Stemming from the point above, if your rocks have 12 material slots, you have done something wrong.
• Use LODs: If your project has zero LOD’d assets, at least auto-generate some LODs for dense meshes.
• Components Are More Expensive Than You Think: This usually shows up in profiling but know that for any object or component that moves in the world, it has to process movement for all its child components. This can be seriously slow, so avoid nesting components and deep component chains if possible.
• VR: Stop Abusing Reflections: Those 20 planar reflections that are capturing in real time with 2048 capture textures look fantastic but will never run at required HMD frame rates.
• VR: Use Forward Rendering Instead of Deferred Rendering: If no one on your team knows the difference between deferred and forward rendering and you are working on a VR project, switch to forward rendering immediately. There are times where I have literally flown to a game company’s office, switched to forward rendering, and then went home as suddenly all performance budgets were met. Exception: If you absolutely know you need deferred rendering in VR, then, of course, use it.
• VR: Use a recent Engine Version: There are a ton of VR improvements every UE4 engine version. You should never be 2+ engine versions behind, you are losing performance for a not very strong reason. Some may disagree, but I have always found the work in upgrading to be worth the performance increase.

Shout Out To All The Engineers Out There

Engineering is sometimes the most asinine and painful thing you can do, but those who love it understand the feeling of solving a problem and having something Just Work. Try not to forget this feeling, especially when you’ve been banging your head against your keyboard for 3 weeks because something that “should be so simple” isn’t.

You are not alone in being alone.

If you haven’t found your personal rubber ducky, please go searching for it.

Michael Allar, UE4 Engineer 

The article was originally published here.