Matthijs Verkuijlen shared some insights into the use of Procedural Dependency Graphs (PDG) in asset and terrain production as well as work with logs and telemetry data.
In case you missed it
You might find these articles interesting
My name is Matthijs (Matt) Verkuijlen. I’ve previously worked at Rare, and before that on Gears: Tactics, Gears 5, Gears of War 4, and Horizon Zero Dawn.
I am currently a Senior Technical Artist but have previously worked as an Environment Artist and 3D Generalist.
For the past few years, I’ve been using Houdini more, and I have investigated PDG use in game dev pipelines. This article will cover some of my learnings.
What is PDG?
PDG (Procedural Dependency Graph) is a newly added feature in Houdini 18. This has added the TOP (Task Operator) context. PDG allows you to distribute and run a lot of tasks, with different settings per workItem. WorkItems can be distributed over multiple cores or can be run in the cloud.
PDG can be used for a variety of things, such as rendering out multiple versions of the same effect with slightly different settings to see which one you like the most, generating different versions of the same assets, applying modifications or data wrangling a series of terrain tiles.
As the name suggests, PDG also tracks dependencies. Changing the settings on a single workItem will allow you to regenerate that workItem and all downstream dependencies without having to regenerate everything else.
For Houdini’s HOULY daily challenge, I built a porch swing HDA. This porch swing had a few input parameters, some of them being the swing’s width, height, seat height, and the angle of the supports. The swing has a little bit of animation, as the seat swings back and forth.
I thought this would be a good candidate to throw in PDG to generate a few different versions of.
I created a TOP network and added in a wedge node and an HDA processor.
HDAs used in PDG processors, especially when rendering things out, could require certain modifications. In this case, I’ve added an Objnetwork and an ROP out network inside of the HDA, in which I added my camera, lighting setup, and render settings. I added a directory and an output filename as HDA inputs and used the wedgenum to make sure it wouldn’t just overwrite the same renders multiple times.
I then set up the HDA processor with the variables I’m generating.
While this is obviously not the best use of PDG, it should illustrate how some of the options can be used.
Using PDG for Terrain
While you could use PDG to generate a whole lot of different terrains, that would probably be pretty resource-intensive (especially when they get larger) and might not get you exactly what you’re looking for. While you could use this to quickly generate a few base terrains for either inspiration or finding something you want to continue with, most games want terrain that is a bit more controlled. Artists and designers usually want to have inputs, terrain needs to be processed, and specific metadata needs to be written out.
PDG lends itself well to split up the entire game world into evenly sized tiles, which can then be processed individually. As long as you split up your inputs, let’s say a road network, in a similar way, you can process these tiles with no knowledge of other tiles.
To continue using the road example, let’s say you have a 10240x10240 meter terrain, and road curves as an input.
Because of the size of the terrain, there might be large areas that don’t have roads in them or entire parts of the world that don’t frequently get modified. Recalculating all their data every step of the way is pretty wasteful, which is where PDG comes in.
If we split this world into tiles of 512x512 meters, we have 20 tiles per axis:
Which is 48 out of the 400 tiles. Especially when you start doing more expensive processes, not having to recalculate unaffected areas can become a lot more beneficial.
Logging and Telemetry
When working with PDG, and especially when you start running your graphs on machines other than your own, getting the right information out of it is sometimes half the battle. Logs get written out to the Temp Directory set on the Local Scheduler within your top network. I would recommend changing this directory to a place where you’ll actually look for logs (and if running on the cloud, a location that can upload these logs so that you have access to them).
After finding the logs, it becomes important to make sure the information written to them is of use. Not all Warnings and Errors get written to the logs by default, especially VEX ones, but all printf statements do.
When your graphs get larger, which can happen when you work on systems like terrain with a lot of data and dependencies, tracking where the time goes becomes very important as well. One of the things I added to every HDA that’s being processed is a timer. This consists of two Python nodes, one at the start of the graph and one at the end, that grab the current time in milliseconds. Since I know when the node started and when it ended, I can calculate how long it took to execute that workItem (note: this doesn’t account for any time spent spinning up workItems or waiting for other nodes, which can add up as well).
Because the number of logs can inflate quickly, and I’d rather not manually sift through hundreds of logs to compare numbers, I built a few Python tools that use some simple regex to filter the important information out of the logs and add it all to one big log. I still have all the information per workItem but now just in a more readable format.
I do a similar thing for the telemetry data, where I go through the logs and use regex to find the lines that fit the way I write my telemetry. I write the telemetry to a json file, which could easily be visualized.
I think PDG is another very useful tool in the Houdini toolbox but it should be treated as such. It’s not a miracle solution that will solve every problem. It’s easy to start trying to solve every problem through the use of PDG but for a lot of problems, PDG is unnecessary or overkill.
Building PDG graphs can be useful to generate a lot of different versions of the same asset, it’s useful for tasks that can be done in parallel, and it’s great to just let things keep calculating overnight or over the weekend.
Before building a PDG graph to solve your problem, it’s always worth asking what benefits you’re looking for from it.
If you want to learn more about PDG, a good place to get started is SideFX’s learn PDG page.
I would also recommend Kenny Lammers’ PDG for Indie Gamedev tutorial collection.
If you have any questions or would like to discuss any topics from the article further, you can most easily reach me through Twitter.