Using Photogrammetry for Real Time Projects

Guilherme Rambelli has shared a breakdown of the latest personal project he did that shows the process of using photogrammetry to create 3D assets for real time use. The guide proves that it’s actually not that difficult to start using this approach. So, study this breakdown, grab your camera and create your first scanned assets for games and real time projects. 

• Key Points to scan subject successfully
• Pre-process image sets
• Solve photogrammetry asset
• Export Mid poly asset, and High poly asset textured
• Create Low poly asset from Mid poly asset
• Clean up Low poly asset
• UV’s
• Extract Maps from High poly asset to Low poly asset
• Bake control maps to modify extracted maps
• Delight Base Color map to create Albedo map
• Calibrate PBR maps (Metallic, Roughness.)

Key points to successfully scan an asset/ Environment

When I go out to scan assets to assemble an environment I usually look for a few things in particular, to ensure what I’m capturing will solve (Photogrammetry 3D result) with quality enough to be used.

Weather – I keep in mind that the flatter the light is when you capture the subject, the better my asset will look after I treat it; The better it will blend and behave in any light condition and scenes. The reason is that it won’t or will have VERY little Highlight/Shadows baked into the Base Color map, which my Albedo Map will derive from.

I try to shoot the images in Raw 32 Bit so there is a little room for error, and the exposures can be equalized later in post.

Rule of thumb

Shoot with the fastest shutter speed possible, keeping the Camera ISO as low as possible to avoid Noise in the image set. Shoot with 30-50% overlap between images to Guarantee that every image shot will collaborate in my 3D asset solve. F-stop may vary according with the lighting, a good range is between 9-13 but is possible that the capture location doesn’t have enough light to support so I just keep in mind those values but don’t compromise the image quality.

* Sharp and Crisp images, F-stop 11 is ideal, ISO the lower the better.

Asset reusability – I always try to capture a limited amount of asset, that would have the most reusability possible, meaning: interesting enough, but not too iconic so people wouldn’t be able to see it repeating throughout the scene.Usually in natural landscapes, the terrain features and “props” tend to repeat themselves quite often, and share common features due to the wearing and weather condition of the location.Sometimes a few good quality captures can take you farther than multiple half good assets.

Pre processing image-sets

There are a few things you can do with your image set before feeding the photogrammetry software with them.There are a couple different softwares out there that can help you deal with extensive amounts of data in a more automated way. My last few year in my adventure to improve my process using photogrammetry I ended up using Lightroom, and here are a the main tweaks I do in every single image set captured and briefly its “whys”.

Lightroom Right Tab, Under “DEVELOP” 1- Calibrate White balance, 2 – Highlight slider set to -100, 3 – Shadows slider set to +100, 4 – Enable Profile Correction (select Lens Mode.) 5 – Under the menu *Photo>Develop settings>Match Total Exposures.

Those tweaks will flatten your image as much as it can with all the 32 bits of color info from the RAW image file. Step 5 will help you fine tuning your image exposure big time, and it will save you some time during capture, adjusting your exposure before every picture.

Once all of that is done, take a look in your image set and see if the color range feels somewhat close to the albedo values you would expect from the material properties from the assets you captured.

Make sure to remove any image your believe is not sharp enough, so it doesn’t influence your photogrammetry solve, and don’t decrease the sharpness of your texture in the next step.

Solve Photogrammetry Assets

There are a several photogrammetry softwares out there, each with its own advantages. The 2 softwares that I had the best results were Agisoft Photoscan, and Reality Capture. The highlights in each software for me were:

Agisoft photoscan

Positive:

• Very forgiven for beginners and give you consistent results even if the image sets are not so great.
• Support 32 Bit images, exports 32 bit textures images
• User friendly and intuitive interface
• Few but solid presets to solve asset
• Batch process with Pro version $+- $350

Negative:

• Takes a long time to process assets
• Asset resolution is strictly limited by the PC specs being used.

Capturing Reality

Positive:

• Extremely fast processing time
• Relatively non limited to PC spec
• Able to generate Results extremely Sharp and detail

Negative:

• Interface less user friendly
• Less forgiven than Agisoft IF image set is not very well captured.

Overall

The steps to generate the photogrammetry 3D asset is quite simple and can be summarized in: Camera Alignment, Rough Cloud, defining solve volume, Dense Cloud, Texturing, Happiness! haha

Export Photogrammetry Data, High Poly, Mid Poly, Low Poly

There are multiple ways to deal with all the data acquired from solving a photogrammetry asset. In this project I used the current state of my personal Workflow, and I hope people can take advantage of it, and can improve it so Photogrammetry can be something more and more common for real time use, and a process more friendly and approachable for anyone.

From Reality Capture, I generate and export the following assets:

• High Poly asset, Eg. this asset generated a 85M tris solve, automatically divided in 60 Chunks of geometry. (Pic 1 and 2 whole and Chunk Geo close up with 1M Tris.) Along with a single 64K texture map.

*Under export options, keep assets into Chunks. so 60 Dif files are generated

• Mid Poly asset, derived from the high poly Yet inside the software Reality capture, all the 85M tris/ 60 chunks were simplified to One 6M Tris Chunk
• Low Poly rough, In Zbrush or other decimation software from choice, Low poly Geo is generated yet presenting several topological problems, mesh handles, etc.
• Clean Low Poly, Once the Low poly is generated, a conventional topology clean up step can be taken in Maya or any other 3D package.
• UV unwrap the Geo accordingly to how it best suit its use. Keep in mind that if you intend to use Tile maps on top of the Photogrammetry asset for blending purposes later in the process, maybe a more fractured UV can give you better results.

Now baking ->

Baking Maps

To bake all the maps Necessary to have full control of the asset and its surface features, two sets of baked maps were necessary during production of an 3D Asset using Photogrammetry.

First maps to bake – xNormal

Once you load all your High Poly chunks, select all the High Poly pieces at once and right click>Base texture to Bake, and select the 64K Map exported from Capturing reality. Once the setup is done, you can bake the following maps.

• Normal Map
• Height Map
• Bake base texture
• Cavity Map

Second set of maps to bake Using 6M Mid Poly asset rather than 85M polys due to performance limitation in Substance Painter

• World Space Normal map
• Ambient Occlusion map
• Curvature map
• Position map

Once all the maps are done, set up the asset backed maps in its slots so the smart masks and filters can work properly in substance.

* Normal Map, Base Texture and Cavity Maps are baked in Xnormal due to allow the load of all the 60 Chunks from the High Poly asset.

Set up Fill Layer as following Base Color – Base color map from Xnormal

Metallic – 0 Value (vary from asset)

Height – Height Map from Xnormal

Setup Input maps baked in Xnormal + Maps baked in Substance Painter.

To start to create the Roughness valuer another Fill Layer was created to contribute with the Roughness values, using the Base Color map from Xnormal, once again, and a Level filter to tweak values and generate a first pass Roughness map.

Cavity map

The first step for creating the first step to delight the base color texture, was creating a folder with it’s mask filled with the cavity map

Folder Mask

The next step was to create a Level to invert the Cavity map values and tweak it so it would mask strictly the areas more affected by shadows/ Occlusion

Fill Layer Mask under Folder

Now that anything in the folder will affect an defined area; a filled layer with a color from choice, picked from the original Base Color map + reference of albedo standard color chart is introduced. Masked by with Smart masks and generators, it adds color variation and helps equalize the color values of the areas once defined as more shadow and occlusion affected

Folder Mask

As a second pass to Flatten the Albedo, I create a folder masked by a Dirt generator to define the areas that needs to be affected by the second delight pass

Fill Layer Mask

The save process is continued with the Fill Layer that lives inside the folder just created. A Filled with a neutral color value is masked by a Dirt Generator and in addition to a Level to introduce a second pass of color values in the areas most affected with shadows and occlusion, flattening the Base Color Map so the asset can better behave in more varied light setups and scenes.

To ensure Texture density in case some areas of the scan has a few blurred pixels and not sharp details.

I used a Filled Layer with a Concrete Substance material, and set the blending mode to Overlay, so an extra pass of was introduced and would guarantee that no texture detail discrepancy would happen if the asset is placed in any scene right next to a procedurally generated texture or so.

To help integrating the assets with each other, I added a soft pass of Dust so a common weathering feature was distributed throughout all the assets scanned.

Adding passes of weathering on top of the scanned asset can benefit the asset, helping it feels like it belong to the same world where all the non scanned assets lived, introducing a soft procedural look to the asset.

Keep in mind that the scanned asset already contains a lot of weathering features scanned along with the asset; so is quite easy to over detail the asset, and losing the real world feel that a photogrammetry asset have.

Photogrammetry Process Specs.

• Image set – 85 Pictures
• Camera model: SONY Alpha ILCE 6000
• F-Stop: 4.5
• ISO – 100
• Focal length – 24mm
• Image Resolution 6000 x 3376 pixels
• Photogrammetry Software – Reality Capture
• High Res Photogrammetry Solve Clean up – Zbrush
• Normal Map, Height map, Cavity map, Base Color map – xNormal
• Roughness, metallic, AO, Final Normal Map – Substance Painter
• Delight Base Color Map to Albedo map – Substance Painter

Guilherme Rambelli, Lead 3D Artist/ Photogrammetry R&D at 8i