Creating a Chinese-Style Vase in RealityCapture, Substance 3D Designer & UE5

Introduction

Hi, I’m Jamie Jamieson, I work as a Photogrammetry Artist. I studied at the University for the Creative Arts in Farnham specializing in Computer Games Arts, which enabled me to really hone my craft and love for 3D environments and 3D Art in general. 

I have worked on a variety of projects, ranging mostly in the film and augmented reality industries. Some examples include; Marvel’s Ant-Man and the Wasp: Quantumania and my most recent gig, The Gunpowder Plot at the Tower of London by Figment Productions.

My role focuses mostly on being behind the camera and creating as realistic a representation of reality in the virtual world as I possibly can. As a person who uses a camera a lot in general, combining my photography skills with my 3D skills seemed like a natural fit for progression in my career.

Getting Into Photogrammetry 

I always had a passion for all things 3D and photography. When I went to uni it became clear that photography (at least at uni) was more of a hobby for me than something I saw myself doing as a career.

When I was reaching the end of my time at University I started to see huge progress in the technology used in photogrammetry and it caught my interest.

At this stage, it was still very early on in its development and produced clunky results. I played around with it and scanned a small rock (kudos to my fellow photogrammetry artists who also started with "the rock"). It looked OK, but nothing I would be proud of, but I could sense that, if the technology was developed more and if it could produce the kind of realistic results that the industry wanted it to, it could be quite revolutionary.

It’s a few years later. I’ve got my head down trying to learn about texturing in Substance 3D Designer when news of a national lockdown hits headlines.

With extra time to spare and a lack of jobs going around I decided to go back to trying out photogrammetry and its new developments. Pushing to learn these new skills with the free time I now had. 

I wanted to test out lots of different textures and objects and needed the space and resources to do this. Lucky for me, I lived next to a golf course, which was closed during the lockdowns. A perfect setting for developing my skills. I scanned different barks on trees, gravel pathways, trimmed grass… you name it, I scanned it. 

Whilst I was scanning and testing out different methods, I also found a few good resources to help me fine-tune my workflow in photogrammetry. I found Grzegorz Baran to be an incredible source of information for learning the first steps of photogrammetry. 

At this stage, I was using a small 12-megapixel Lumix DMC-G1 camera from 2008 (the minimum recommended size for shooting for photogrammetry is 24 megapixels) so it was an unideal setup and I didn’t expect much from the initial scans. Then, the results started coming in. I was astonished at how decent the results were and to this day, some of these initial scans still remain in my portfolio!

These initial scans motivated me to keep testing more textures and objects and I took my scans indoors.

What I find fascinating most about photogrammetry is it’s the real deal, it’s almost an exact replica of an object or ground right in front of you, and now you get to see it in 3D and throw whatever lighting you want at it. It’s this that has motivated me to scan more and see how far I can push my skills and the craft itself. 

To summarise, what motivates and inspires me the most about this craft is the sheer authenticity of what you’re sharing in the digital world. You can’t fake what's real, and photogrammetry is about as real as it currently gets.

The Chinese Vase 2 Project

One of my latest projects, Chinese Vase 2, is the second vase in a series of – you guessed it – Chinese vases.

To capture the initial data for this specific project, I used the far more powerful Sony A7RIV (61 megapixels!) which made the results a lot cleaner. I have a lot of old Chinese objects about the house from when I lived in Hong Kong, so I thought it would be interesting to go through some of them and try and scan them. I initially took about 1608 shots in a Bracketed HDR format (563 photos after exporting from Lightroom combining the 3 bracketed photos into one), which allowed me to have a greater range of information when it came to this very reflective object (note, I didn’t have a cross-polarizer setup which would have removed most, if not all, reflections).

Without a cross-polarizer, this technique was absolutely necessary to mitigate any potential problems. From my experience when it comes to reflective objects it's always better to take more photos than you think you need. Kind of like insurance – you don’t think you need it until you need it. I try to imagine what the worst-case scenario could be before it could happen. And the last thing you want to do is have to take photos again. Trust me, I’ve done it.

For my lighting setup, I had a basic three-point light set up of LED panel lights, two in front horizontally to the object, and one at the back slightly raised above the object. I used a simple green screen as the background which helped Reality Capture to focus on the main subject and "ignore" the rest. I also had a simple Turntable setup, nothing fancy, just a simple spice lazy suzan that I could manually turn with my hand.

Processing the Data with Reality Capture

Processing the data with RealityCapture was relatively straightforward. Once the capture and lightroom phases were completed, I simply exported and dragged-and-dropped the photos into RealityCapture and pressed "Align images".

Sometimes you’ll get a few cameras that just don’t understand what’s going on, which is what happened with me. Luckily, there were only a handful of cameras that went astray for me. I added in a couple of control points which re-jigged them back into their correct positions and could move onto the next phase.

It was now time to build and clean the mesh. With other projects, this can take some time depending on the photos and what level of detail you’re aiming for. For this specific scan, given it’s a highly reflective object, all I needed was the silhouette of the mesh. The fine details were jumbled about, so a "Normal Detail" setting was all I needed to do this phase.

When the mesh was built, it was time to build the texture. I used the EXR file format, which allowed for a greater range of color. This can be useful if there is any rogue light or shows you need to even out. On this project, the texture was clean so further editing was needed.

RealityCapture's Strengths

I think RealityCapture’s main strengths are that it’s fast compared to its competitors, it’s straightforward and easy to use, it’s reliable, and it has a whole host of features from drone to LIDAR that I hope to be able to one day use. Knowing that these features are there as an option is great.

As for the current state of RealityCapture, let’s just say I’m really happy with using it now, it does a great job, and I'm looking forward to seeing where it goes with the advances of technology. Hopefully one day I won’t have to use control points quite as much!

Texturing

The texturing process begins when the object is generated. In this project, when the vase was generated it came out with a few bumps on the mesh because the vase’s surface wasn’t smooth. RealityCapture was trying to add that detail for me. The reflections in the vase confused RealityCapture’s interpretation of the shape in the places that had much more detail, which resulted in these bumps. All I needed at this point in the workflow was for the object to be smooth. I would add the finer details such as grooves later. 

Next was the cleanup phase which was a simple process of importing the high poly model into ZBrush, smoothing out the bumps, and creating a simple low poly version to work with. 

It’s important to export your object with the same transformations as the original RealityCapture model. Transferring the color texture will be significantly more difficult if you do not do this (you’ve been warned).

After exporting my vase, it was then just a case of retopology in Maya. I used a curve to trace an outline of the vase and then a loft tool to get the base shape. I then converted it into a polygon object.

In some cases, with this technique, you might need to bring it back into ZBrush and project the model onto the high poly if it's clay sculpted (which is what I had to do with the vase as it's not 100% symmetrical on all sides due to it being handmade). The UVs are pretty straightforward, just a simple seam down the side as well as on any hard corners. 

Then, all I had to do was transfer the color texture to Substance 3D Designer and work on creating the other maps from this. This involved making a copy of the colour texture, making it black and white, and creating the roughness as well as the other maps.

The variety of colours on this vase meant that I had to target certain colours with different properties when it came to the roughness map. I used Photoshop to manipulate the black-and-white version of the colour map to best represent that detail. Then using another black-and-white map to create the normal map. There were obvious seams in some areas of the mesh where a UV border was, so I used Substance 3D Painter to simply clone over these areas. 

Presenting the Final Mesh

Presenting the final mesh with all its details was the next phase of work.

I sent the final mesh to Unreal Engine 5 for presentation. Unreal Engine 5 works fantastically well with Lumen and Raytracing. The process of having to gather a scene together with light maps and other technical aspects is mostly removed when using Unreal Engine 5 so it’s a real timesaver.

To present this, all I wanted was a simple studio-style setup, nothing technically fancy, just a solo product showcase. The talented Joshua Lynch has helped me in the past with portfolio presentations so I used the knowledge I gained from his mentorship and applied it here. This was a simple three-point rectangle light setup, with a dark backdrop and a simple prop from Megascans for the vase to stand on. The cinematic camera was also a simple setup, just a 16:9 film setting with a simple depth of field. 

The post-production was a straightforward one with a few minor tweaks here and there. To set up the scene I used a simple "Actor" to fix the camera so I could have more control of the camera direction which can be helpful if the origin point of the object is not at the center of the mesh as the camera will track the origin. 

At this point, I applied an LUT to the scene. I decided to experiment and try using Lightroom for this with small adjustments to the contrast, curves, and saturation. 

I looked at a lot of product photography on Instagram to gain some inspiration on what kind of mood I was aiming for in the scene (go follow Peter McKinnon for some inspo!). For the render settings, I combined the advice of YouTube personalities William Faucher and pwnisher to get the results I wanted. I then exported the render in the EXR format with 24 FPS to give it a more cinematic look.

The result was a simple spin around the vase hooking the cinematic camera to a rail and animating it, with a little camera shake added. At this point, you can use whichever editing software to combine the EXR files into one long video. It's down to preference as to whether the user wants to add any special effects or not in an external video editing program, for me, I like to try and keep all the post-processing within Unreal. 

My main tip to anyone thinking of presenting their photogrammetry work in Unreal Engine 5, would be to look at photographers' Instagram pages that showcase objects. Since you’re showing off a real-life product within a 3D engine, it makes sense to take inspiration from photographers. Another tip would be to keep your scenes simple. If you're showcasing a single object, most of the time you don’t necessarily need an amazing scene in the background of the main subject, otherwise you might as well showcase that too! So keep it simple as if you are in a photographer's studio. 

The RealityScan App

The recently-released RealityScan app offers an amazing way to get into photogrammetry at a beginner level. I think it will certainly help people get some base meshes to work with. 

I don’t have an iPhone (don’t judge) so I can't use the app. When it comes to production use, I can certainly see RealityScan being used by producers/directors to get location ideas put through or even used by Independent filmmakers to help with production without investing in heavy-duty equipment.

If you want to have more advanced/accurate scans, however, a heavy camera will be the one that will get you those results. RealityScan is certainly a good thing for people to get an understanding of how photogrammetry works on a basic level.

Tips for Photogrammetry Beginners

As for tips for beginners, the first thing to consider is what gear you have, do you have a decent camera or just a mobile phone? Both will require a different approach when doing photogrammetry. 

For mobile phones, RealityScan will help guide you most of the way. The main thing to consider then is your lighting situation. Ideally, you want it to be an overcast day, this is easier when you need to remove the ambient occlusion. If you are an android user like me (again don’t judge), RealityCapture will allow you to import a video and use the frames to build the point cloud. Another useful thing to have is a ruler or object that you know the size of. This will allow you to resize the object to a more accurate size in your 3D program of choice. 

If you have a decent DSLR or mirrorless camera you'll need to invest in the basics. That means you need a tripod, a polarizer, and an external trigger as well as a colour checker and ruler. The colour checker is your best friend, use the grey/white side to set your exposure and the colour side to get the true colours of the scanning area. 

As mentioned before, the best way to shoot photogrammetry is in an overcast environment and the first object you probably want to scan is a rock. Pretty much everyone starts off with this, rocks have a lot of detail and it's a great object to get to grips with scanning. Make sure you take your photos in RAW at as low ISO as possible, a maximum of 400 ISO, ideally 100 ISO. 

How you scan is dependent on the object. For a surface area, you'll want to go from side to side moving up slightly as you do, for an object you'll want to spin around in a circle, I prefer starting from the highest point and ending at the lowest point. A good overlap between photos is necessary, you'll want 80% of the previous photo to get an accurate result and mitigate any potential errors. This is the technique I used when I started out with my 12-megapixel camera. 

The lightroom phase is pretty simple, push the highlights down and bring those shadows up to remove as much AO as possible. Then use the Colour checker to get the correct white balance and the X-Rite tool to get the accurate colours. And then it's time to export, PNG/is are generally ok, but TIFF I find too large a file format and eats up all your hard drive space unless you have true 16bit or higher images. At that point, you'll be using EXR format, but that's a more technical area which I won't go into detail about here (you'll be reading a book, not an article). 

RealityCapture like Unreal Engine 5 is pretty fantastic from the get-go. The main advice I would give when handling RealityCapture (or any photogrammetry software) is to make sure you have enough photos and that they are the best quality possible.

If RealityCapture splits some photos away from the main pack, that generally means something is confusing RealityCapture. You'll need target points to align the stray photos to the main group. The RealityCaptures YouTube channel has great tutorials for working with these. 

To summarise, take the best photos you possibly can – it all starts when you are outdoors or in the studio – cover as many possible problems as possible and you'll be getting pretty decent with whichever gear you choose to use! 

Thank you for taking the time to read this article on 80 Level! Follow my art at ArtStation.

Jamie Jamieson, Photogrammetry Artist

Interview conducted by Theodore McKenzie