LogInLead Photogrammetry Artist at Rebellion Kristóf Rosu has told us about the team's photogrammetry workflow, talked about creating assets for Sniper Elite 5, and explained how they capture tall objects.
Intro
Environmental photogrammetry at Rebellion is a massive undertaking. In the last couple of years, we’ve visited close to a hundred locations, scanned thousands of objects, and generated hundreds of terabytes of data. It all started very small though.
After the release of various titles taking advantage of the technology, I and a couple of other artists went out on a lunch break with a camera to see what all the fuss was about. The first tree stump we captured made it into multiple games and kick-started the process that led to Sniper Elite 5 being based almost entirely on photogrammetry assets.
Shortly afterward we started to do more scans, develop our kit, and even go on some trips. You might have seen some of these early assets in the DLC for Sniper Elite 4, then a lot more in Zombie Army 4.
For Sniper Elite 5, the decision was made to source the majority of assets from scanned data, so a dedicated team was established. We have seven artists, a dedicated location manager, and together with the outsource team, we make up Rebellion Digital Assets. Here are some examples of our work:
The whole process starts with locations. Ross, our Location Manager, puts it like this:
"Scanning in the real world needs preparation. The goal is to be granted complete access to an asset with commercial image rights lasting well into the future. It is certainly good practice for photogrammetry companies of all sizes to avoid ‘guerrilla scanning’ at any level. We are proud to have developed good, open relationships with organizations such as the National Trust, and the Imperial War Museum, as well as British councils, French mayors, private owners, and estate agents like Carter Jonas. And as such, doors have been eagerly opened, giving us the opportunity to put into 3D form assets such as the Sherman and Tiger II tanks, swathes of beautiful Lake District, an 18th-century water mill, Normandy architecture, the IWM’s V2 rocket, sailboats, pubs, trains, helicopters, and a PAK-40 used in Band of Brothers.
Our job takes us to places that often offer stories that chime wonderfully with the games we’re making. Military academics, estate managers, and even WWII-era Frenchmen add to the color and detail of our scans and often we find ourselves being pointed in the direction of the perfect asset that we had never thought available to us.
All this comes from good, solid research and groundwork before the first photo is taken."
The mission of our team is to scan almost everything that goes into a game, and once you move on from photogrammetry’s most ideal subjects – rocks, tree stumps, brick walls – things get complicated. Vehicles, buildings, furniture, and foliage – all require unique techniques and equipment. We collected some of our best tips and ideas and decided to share them with you.
The Rig
Our basic kit is fairly standard and you might have seen it used by other photogrammetry outfits. It consists of a high-resolution mirrorless camera, a zoom lens, and a ring flash with cross-polarised lighting.
Here comes our first tip – these Godox ring flashes that everyone uses are fairly notorious for their battery – it’s hard to find spares, they break easily and don’t last very long. During the worst of the battery shortage, we decided to retrofit all of our flashes with external batteries. You’ll need a bit of soldering knowledge for this, but it’s definitely worth it. For the external batteries, 12V and 15A are the requirements. We decided to just remove the cells from an original battery and attach the wires from the external pack directly to the pads. This has the added benefit of moving some of the weight from the hands onto a belt pack.
Scanning Big Objects
As production of Sniper Elite 5 progressed, we were required to scan larger and larger objects such as buildings, cliffs, trains, trucks, and even an entire tree. We needed to get our cameras high up, into position.
Our first idea was to use a drone, and we still do when absolutely necessary, but it wasn’t an optimum solution. They are cumbersome with lots of regulation and paperwork to deal with, the image quality is not comparable to our cameras and we couldn’t mount a flash on a reasonably sized drone. Another solution was a telephoto lens, which we’ll talk about later.
One of our artists, Jay, talked about what we came up with in the end:
"The method we use to scan tall or out-of-reach objects is pretty unique in the photogrammetry space. One can already imagine the difficulty that arises when trying to scan 20m tall buildings or trees and large trains or tanks. To capture the full dimensions of these assets we needed to think outside the box: long, carbon fiber telescopic poles, originally meant for cleaning windows.
There are three important factors to take into consideration when choosing the right pole for photogrammetry: the overall weight of the pole and equipment mounted on it, the maximum height it can reach, and its sturdiness. Our telescopic poles are made out of carbon fiber which is a strong and lightweight material. The camera, our custom ring flash, and battery were all mounted on top of a ball head attached to the pole.
Adding everything up, things get very heavy and risky! This particular way of scanning requires at least two people to counterbalance the weight of the camera when reaching higher levels. Using two different sets of poles, with maximum heights of 8m and 19m, we were able to capture most of the surfaces we wanted to scan. We use a wireless remote to trigger the camera and when the pole reaches higher levels, we also use an iPad to visualize where the camera is pointing."
"Going into further detail, the scanning patterns we use depend on our subject. The taller it is, the more height increments the pole has to be extended to. We capture each height level in one go to maintain an evenly distributed pattern with plenty of overlap between each shot, then raise the pole and repeat."
These days we take the poles on every trip. Even when we work inside, in a museum, for example, we can use them to get good angles above larger objects like tanks, artillery guns, or tall furniture.
The Photogrammetry Workflow
For photogrammetry to work, it is essential to have ‘features’ on our subject to get a clean scan. These can be anything from the natural detail of a rock or tree bark to dust, mud, or scratches. They are used by the photogrammetry software (RealityCapture in our case) to both align the photos based on their content and to generate depth maps for them by comparing images in pairs.
Unfortunately, a lot of our subjects don’t have this kind of surface detail, especially the vehicles and other military hardware we often have to scan. We’ve come up with a few solutions over the years, like taping beer mats or more professional coded targets to problematic surfaces, using a sublimating scanning spray from Aesub, and even laser scanners.
By far the most memorable one happened when we went to scan a Pak 40 anti-tank gun. It was stuck in the mud, and it took all of us on location to get the cannon, and the forklift sent to rescue it, back onto solid ground. As it was already very dirty, we got permission to spray some more mud on top. This real-life texturing process not only provided the perfect surface detail for the PG software but also made the final asset a better fit for the muddy battlefield.
Another unusual method of capturing large objects is using telephoto lenses. If our subject is too large for the poles, or we can’t use them for another reason, we grab a couple of 400mm zoom lenses. These let us reach our minimum texel density from much further away, thus allowing for a less oblique camera angle to our subject.
The shooting pattern has to be adjusted a bit. Parallax between different images is essential for photogrammetry to work well. Unfortunately, the longer your focal length, the less of this parallax effect you get. The best pattern we found is to shoot ‘panoramas’ from each location that encompasses the entire subject. This effectively simulates a more standard field of view, but with a much higher resolution.
It’s not enough to just capture the data, it also has to be processed. In this regard, we found the automated solutions to be inadequate for more complex scans, so we tried to create a fairly traditional, but highly optimized pipeline instead.
We worked with our IT department to set up a large storage server and remote processing machines all hooked up to each other with high-speed connections. We even created a dedicated ‘batcher’ software that distributes various RealityCapture processes to our available remote machines. This automates the initial, boring parts of the process so artists can focus on scanning and creating game assets.
After we have our high-res scans, we assign these to an internal artist or outsource them with the help of the OS team. This part of the pipeline relies on all the usual software – sculpting tools, Substance 3D, Photoshop, and a special shout-out to xNormal, which allowed us to bake our largest scans, with 64k textures and close to a billion triangles.
Conclusion
Thank you for taking the time to read this and I hope you found it interesting. If you did then it’s worth noting that we are recruiting! We are looking for an experienced artist to join our team. If photogrammetry is not your thing, we also have many other roles open across all disciplines. Head over here to check them out.