LogInWe were lucky to talk to the amazing team behind a popular online shooter Escape from Tarkov and discussed the amazing technology, which powers this product.
We were lucky to talk to the amazing team behind a popular online shooter Escape from Tarkov and discussed the amazing technology, which powers this product.
Introduction
At the moment, our team has built up to almost 70 people in-house and over 30 on the outsourcing. Level of experience is different, but we don’t have any of the so-called rockstar developers of art directors from renowned studios. Everybody on the team more or less learned and evolved with us.
While Tarkov is partly a Unity game it looks nothing like a Unity project. I’m wondering if you could talk about the main advancements you’ve made with this tool and what are the key additions you’ve added to the render and maybe some other systems of the engine?
Indeed, we have put in a lot of effort to make it look like that. While developing your own shaders is pretty common, we also did post effects and created our own tools for lighting, correct rendering of indoor and outdoor environments. Also, many different tools were prepared to facilitate working on environment and level design which saved a lot of time. It’s also worth noting that good quality of assets is very important – weapons, gear, items, props, everything.We paid a great deal of attention to them from the start, and such approach has paid off. They will, most likely, look good in any engine, though, of course, we did some tweaking to maximize that impression.
What does your production process usually look like?
In our production process we employ or at least attempt to employ all the innovative means there currently are in the market. We used World Machine as well – to generate the general Tarkov and suburb terrain. Scanned objects too – huge thanks to Quixel Megascans who granted us access to their library, they are awesome and we appreciate it. And of course we scan some peculiar and unique items ourselves. In general, we always try to enhance the production with something new, and we are always open to ideas from anybody on our team on how to make the pipeline more artful, smart and efficient. All sensible suggestions get researched, tried and tested, we are very flexible in this aspect.
When building these amazing big spaces how do you approach it?
Great question. We assume that we’ve employed a rather thorough approach to location design. First of all, we’ve planned out the whole region, down to the finest details possible at such a scale – terrain, buildings, infrastructure, minor service facilities and landmarks. Then, we have divided it into raid maps which we are gradually preparing now, fleshing it out with details. The workflow for particular location starts with rough sketches, transferred over from the region map. Then, the architect takes over, adjusting everything for realism and adequacy: placing buildings, roads, service stations, pulling in power lines and pipes, setting notable objects, etc.
Next step is to block it out – we draft the objects, play the map and test everything for physics and visuals (yes, even at this stage). And while the draft the objects – which will already be playable when inserted – we give the long huge prop lists to our outsourced team, who are producing the particular objects we’ll be placing on location. After that, we start processing everything by appointing several key level designers, each responsible for his own area of the map, who place all received items in the designated area, making it look alive. The last finishing touch is usually on me. I do the additional optimizations, environment art, revise it and remove occasional bloopers, make minor adjustments – generally speaking, polishing and making it look beautiful. And now, as we are in production of the next upcoming location, we offer the maximum creative freedom we can, for our guys to try and make everything by themselves, without too much pressure and supervision, and such approach has paid off, yielding some interesting results. There are, in total, up to ten level designers and environment artists working on locations. Ah, almost forgot one important thing: at the very start, when we only have a rough sketch, we give it to our concept artist who draws it in detail, setting the general look, mood and ambience of the place, and drafting a part of the prop list while at it, thus helping environment artists and level designers to recreate the same mood in 3D.
One of the key features of the game is the amazing authentic animation system of weapons and the whole way the weapons work, with all the misfires and other details. Could you tell a little about the way your modeling guys work with your animation and physics engineers?
Right now, at this moment, there is no especially tight communication between modeling artists, animation designers and engineers. Right from the start we have developed a great entirely procedural physics system to deal with weapon handling, recoil, wear etc. There is a long list of adjustable parameters covering all aspects of weapon behavior we could think of, and, of course, all of them are visualised through an extensive set of animations, which were tailored for specific weapons – visual examining of weapon, folding and unfolding stock, operating fire mode switch, checking of ammo, jamming, stovepipes, etc. All of that is done by people who know how weapons work, and, most importantly, have experience operating them. They are regularly shooting at the range, and have a good idea of how it feels like to shoot. I think it’s a really important factor, and would be an essential asset for any realistic FPS developers. Knowing and feeling weapons firsthand really matters.
Could you tell a little about the way you make the necessary optimizations here? How do you work on draw distance, dynamic level of detail, imposters and other stuff?
Our game is CPU-oriented. Though we are constantly monitoring the performance and running the necessary optimizations, it’s the CPU that carries the weight of the animation systems, and gets very busy at times, more so than GPU. Graphics gets optimized uncompromisingly. My eight years of experience have left me with a big set of tricks and techniques to optimize the heavily loaded locations. We work with LODs, imposters, strictly contain draw calls per frame and otherwise try to keep within allowed limits. The main huge problem right now is textures, we have a lot of them, and in very high quality, while the Unity version we’re using now doesn’t have any texture streaming. That’s why the game consumes an immense amount of RAM with high-quality textures settings. Of course, that will be optimized as well, most likely, we’ll be adding our own streaming system. We do frequent preventive profiling both with standard Unity means and third-party tools to discover any possible weaknesses, abnormal surges in consumption of CPU and GPU resources, RAM and VRAM. Sometimes, quite regularly, in fact, we put the active development on hold to do the profiling and optimization interludes, and then I personally work nights to dilute the level of detail, combine objects together to reduce the number of draw calls, stitch textures into atlases to reduce their number, simplifying shaders… Basically, we employ almost all means currently available in Unity, and, naturally, try to think of something of our own, for instance, scripts that exclude the unused objects from rendering in semi-automatic mode, sort of occlusion culling for the poor. Unity’s native occlusion culling is rather heavy and puts quite an extra stress on CPU, which is already pretty busy.
Probably the most effective approaches are LOD and textures optimization, cutting down the number of objects and reuse of objects. It often happens that somewhere on location far far away there stand a coffee machine with 1k textures, 3 of them, normal, specular, and diffuse. It seems like a trifling matter to remove it, it will only free up 10-15 MB of VRAM. However, if you spend the night doing it and find a score of such objects, that’ll top up to 100-200 MB. A week or so at it will free up 500-600 MB more. So it goes, that’s how we optimize, a rather tedious process. That’s why it is important to always keep that in mind at the initial level design stage. All level designers need to understand and fully realize that every new prop added to the map must be reusable as much as possible, otherwise it’s useless and will only consume valuable resources.
How did you work on the lighting in your project?
We took the pains of creating our own lighting system. Basically, everything outdoors is lit with ambient light with spherical harmonics, i.e. we take into account sky lighting, everything is true here. For indoors we’ve created, by using stencils, special zones that tone down the ambient light, and created portals that let the ambient light through. That’s how we achieved the effect of indoors being dark and properly lit with ambient light from the outside. As for dynamic lighting and shadows, we work with the standard ones Unity provides, nothing we could do about them. Of course, we cut their distance and use separate, simplified objects for shadows, they have less polygons and vertices, thus slightly reducing the total number of polygons in frame. Otherwise it’s the same. E.g. Unity does not have settings for particular light sources where you could possibly set the shadow draw distance from that source. Unlike CryEngine, Unity doesn’t allow that.
Overall, what do you consider to be the biggest problems you’ve encountered during this production?
The greatest challenge, after all, is network and network infrastructure. Graphics and performance optimization is a hard task indeed, but it already has a solution. There are parts in graphics pipeline that can be enhanced to make it turn out well. Network, however, is harder. We write a lot of things of our own, of course, and encounter a huge amount of issues. Needless to say, bad network-related components can seriously damage gameplay, it’s a core-breaking thing we can’t allow. That’s why it always puts a lot of pressure on us in cases when, for instance, we’re launching an update with network optimization and something’s wrong with it. There were many occasions like that when it comes to network and it’s always hard. Naturally, there are also bugs and other tech issues… Not sure if I can single out something in particular. Almost every month, some terrifyingly humongous problem arises. How do we solve them? We get down to do our research and look for possible solution, and there always is a solution or workaround, nothing is impossible if you try hard enough. Moreover, Unity is helping us too, support sometimes aides us with some questions, and we hit the roadblock, we often develop in parallel, if something can’t be done with a specific function, we devise a workaround. That has its price, sometimes in performance, but in the end it pays off. Our project is not ideal in terms of tech, it’s really large and complex, it has a lot of parameters that can misbehave and a lot of components that can possibly fail due to its very complicated architecture, but we are gradually improving it, making it better, adding new features and repairing old ones. It’s a hard way, but we keep our spirits high with realization that we are the innovators here and no one has done something so complex in Unity before.