Defining a Displacement Shader
The displacement shader is a powerful tool to add realism and detail to your renders. Displacement mapping works by adjusting the elevation of a model’s surface using a grayscale texture: brighter values push vertices outward, darker values pull them inward, and mid-gray values have no effect. This method differs from normal mapping and parallax height mapping, which are pixel shading tricks that create the illusion of depth. In contrast, displacement mapping physically changes the geometric shape of the model.
Displacement Shading in Toolbag
To enable displacement, click on your material in the Materials window, look for the Displacement module, and choose a shading model type from the dropdown.
Upon enabling Displacement’s Height model, you’ll find several sliders to adjust.
- The Scale slider controls the intensity of the displacement effect, determining how far vertices are pushed or pulled from their original position.
- Scale Center calibrates the displacement’s zero-point, setting the surface’s baseline or starting position before displacement. This ensures the effect originates from the intended midpoint, which is adjustable via the slider.
- Contrast exaggerates differences in the height map, with higher values amplifying the separation between low and high areas.
- The Contrast Center sets the midpoint of contrast adjustments, determining which gray level is treated as the central height value. Adjusting this biases contrast enhancement toward either highlights or shadows, helping to emphasize specific details.
- The Offset slider shifts the overall displacement data up or down. For example, if sharp edges are causing crumpling in the base mesh, adjusting the offset can realign the displacement effect, reducing distortions.
- Enabling Generate Normals will update the mesh normals if a normal map isn’t available, ensuring the displaced model shades correctly.
- When Relative Scale is enabled, displacement is applied relative to the size of your model’s bounding box instead of using an absolute, scene-wide unit scale. This is particularly useful for ensuring consistent displacement across models of varying scales.
Displacement is particularly effective when combined with Toolbag’s mesh subdivision feature. Before we continue learning about displacement, we must know a little about subdivision techniques.
Subdivision, or Sub-D, adds predictable tessellation to a model while preserving its overall volume. This is especially useful when more polygons are needed in specific areas to support displacement effects. Higher subdivision levels allow more detail to be extracted from a displacement map, though this comes at a performance and memory cost.
Learn how to subdivide meshes in Toolbag.
Toolbag’s Subdivision Modes
Toolbag offers three subdivision modes: Catmull-Clark, PN Triangles, and Regular.
Catmull-Clark
Catmull-Clark progressively smooths a model by softening sharp details and polygonal edges. Each subdivision level exponentially increases the polygon count, so starting with a low- to mid-poly mesh is recommended. The Sharpen Corners option preserves the edges of open geometry.
PN Triangles
The PN Triangles mode smooths triangle-based surfaces, turning sharp, flat areas into more natural, curved shapes without significantly increasing the polygon count. The Smoothing slider adjusts the smoothing amount applied to the model.
Regular
Regular subdivision divides polygons without changing the model’s shape. It’s ideal for quad-based, hard-edged models not prepared for subdivision. Each subdivision level increases the polygon count by a set factor, so starting with a low- to mid-poly mesh is recommended.
Preparing for Subdivision
Sometimes, you may need to add edge loops to your mesh in your modeling app in preparation for subdivision. For example, if you want to preserve important hard edges or creases while using the Catmull-Clark subdivision mode.
It’s easy to switch between subdivision methods in Toolbag, making it simple to find the best option for your model. Adjusting the Subdivision Level helps optimize performance by keeping scenes lightweight and quick to load until additional detail is required, such as for a final render.
Isolines are enabled by default to show the original wireframe edges adjusted for subdivision while hiding newly subdivided faces. This prevents the viewport from becoming overly dense when viewing subdivided meshes in wireframe mode.
Not every model requires subdivision for displacement. Still, it’s very convenient to control detail in Toolbag directly, eliminating the need to return to your modeling app to add detail and re-export. Toolbag’s real-time controls make it easy to determine when the appropriate subdivision level has been achieved.
Toolbag’s Displacement Shading Models
Toolbag has three displacement shading models: Height, Vector, and Parallax.
Height
Height adjusts elevation along vertex normals, pushing or pulling vertices without directional data. Ideal for brickwork or height-mapped terrains, but cannot create overlapping features like caves. Various sliders for scale, depth, and contrast allow precise displacement adjustments.
Vector
Vector uses magnitude and directional data to determine how far and in which direction vertices move from their original position. This allows for complex details that height maps cannot, such as overhangs, caves, or tunnels. Vector displacement maps can be created in software like ZBrush or Blender but have limitations.
Parallax
Parallax is a pixel-based effect that simulates depth without altering geometry, leaving silhouettes and shadows unaffected. Driven by grayscale height maps, it works well for surface detail but is less suitable for significant elevation changes than height or vector displacement.
Displacement Format Recommendations
Displacement maps can be generated in various ways using Toolbag’s Bake Projects or other software, and in many texture formats and bit depths. Ideally, displacement textures should be uncompressed, and image formats such as .tga, .tiff, .png, or .exr are recommended.
32-bit EXR
Faithful representation of original.
16-bit PNG
Less precise, as seen in the cone’s curvature.
8-bit TGA
Displays some surface banding.
For simpler displacement scenarios, such as bricks or cobblestones, 8-bit textures are usually sufficient. 8-bit textures can be stored in the alpha channel of an RGBA texture or in a packed mask texture (AO, Roughness, Metalness, Height), which can save memory. In some cases, textures with lower bit depths can introduce banding artifacts, which can be mitigated using a 16-bit texture or higher.
For more complex displacement setups, such as terrain, clean mechanical shapes, or fine organic detail, 16-bit or 32-bit displacement textures are recommended, but they will have increased file sizes and memory use. Uncompressed texture formats include .tiff, .psd, and .exr and are recommended for these scenarios.
Learn how to use Bake Projects in Toolbag.
Correcting Common Issues
When applying displacement mapping to a mesh, you might notice that the mesh normals don’t shade as expected. Typically, a normal map is used alongside a height map to address this issue, or Generate Normals is enabled if a normal map isn’t available.
Displacement Without Normal Map
Displacement With “Generate Normals”
Displacement With Normal Map
Mesh topology plays a critical role in achieving clean results with displacement mapping and subdivision. In the comparison image below, the cube on the right demonstrates evenly spaced topology, while the cube on the left shows irregular topology, leading to less predictable results.
Drag the handle between the images to compare the results using bad topology vs. good topology.
When each cube mesh is subdivided, additional triangles are added to both, but the base mesh topology determines where the subdivisions occur. On the cubes with uneven topology (left), the large triangles fail to subdivide as effectively as the smaller ones, resulting in less detailed displacement. In contrast, the cubes with even topology (right) subdivide more practically and predictably, allowing for uniform displacement. Notice how the uneven topology on the right causes the cube to warp and pucker in areas lacking sufficient triangles, leading to visible artifacts and shadow issues.
Baking Displacement Maps in Toolbag
Let’s explore displacement texture baking using Toolbag’s Bake Projects and Interactive Baking. This process can accelerate your workflow by generating a height map from a highly detailed source mesh so that a lower poly model can be displaced to match it while retaining the subdivision. In the following example, we will create a sand pile using a source mesh generated in Gaea and a simple plane for the low poly base mesh.
Set the Bake Project’s Output Format to 16-bits/Channel, then hit Configure under the Maps rollout and choose your textures – Normals, Height (Displacement), and Ambient Occlusion will be sufficient for this bake. Increase the default resolution (2048×2048) if finer detail is needed. Finally, align the models, positioning the green low-poly plane beneath the high-poly mesh.
Select the Low Bake Group, then open the Cage menu to adjust the Cage Max Offset. Ensure the cage fully encompasses the high poly to avoid cutting off any peaks. Here, the offset was increased from 64 to 95 to provide sufficient coverage.
While baking, progress is shown in the lower right corner of the viewport, and any changes to the scene will trigger an automatic update to the textures using the Interactive Baker, which is useful in displacement mapping as it displays captured height data in real-time while adjusting parameters.
Hide the High Bake Group to focus on the baked results and normalize the data via the Height settings panel (⚙️) if the preview appears overly white. For best results, set inner and outer distances to equal but opposite values (e.g., 5 and -5), centering the midpoint at 0.5. This is especially useful for the displacement of characters or props.
Adjusting Inner/Outer Distance clarifies the height range, with peaks shown as lighter areas. Using a flat plane as the low poly mesh highlights the entire height range, as the difference between the low and high poly models is substantial. A closer-matching low-poly mesh would yield a less pronounced height range.
Applying Baked Displacement Maps
Set the Bake Mode to Offline to disable the Interactive Baker and prevent automatic rebakes from displacement or low poly mesh edits. Load the baked Height map by clicking the checkerboard texture under Displacement and selecting it from the output directory. Since the low poly model lacks sufficient detail for proper displacement, select it from the Scene and enable Subdivide in the Subdivision menu.
Increase Subdivision Levels gradually, monitoring triangle and memory stats. Adjust the Scale parameter in the Displacement low poly material settings to combine subdivision with displacement. Refine any more displacement settings as needed in the Displacement material module.
Combined with albedo, roughness, and normal maps from the integrated Library’s Sand Uniform material, we now have a versatile and scalable subdivision asset without leaving Toolbag.
Learn more about Toolbag’s Library in our Library Guide
Methods like this are widely used in the film and broadcast industry, where high-resolution detail is important, and normal maps alone are insufficient. This is especially useful in more complicated pipelines (i.e., character work) since animators can work at lower subdivisions for rigging and animation while surfacing and rendering teams can work at higher subdivisions with displacement to restore detail.
Displacement Using Material Layering
Using layered materials in Toolbag, we have even more control over displacement effects. It’s possible to combine displacement maps with varying heights and detail, allowing for granular tweaking over how they are blended together.
In this example, I’ve built upon the sand pile example from the Displacement section above and added a second displacement material that respects the original height values rather than completely overwriting them. This is achieved using the “Overlay” blend mode, which takes the light and dark pixels of the topmost layer and merges them with those of the bottom layer, preserving mid-tones. At the same time, highlights and shadows become more intense without losing detail. This blend style allows us to control the overall height of the pile in the base layer while separating controls for individual rock displacement in the second layer.
Seamless Displacement with Triplanar Projection
Using displacement in Triplanar Projection mode unlocks some interesting use cases, such as blending arbitrary models together or kitbashing. It’s possible to compose complex shapes in the viewport by placing meshes so that they intersect and using triplanar displacement to apply cohesive, unifying detail across surfaces, regardless of their UV layout. Seamless detail across surfaces is possible when different meshes have similar subdivisions and topology.
To use triplanar mapping, go to the Texture panel of your material and change the drop-down from UV to Triplanar. Apply this material to your models, and as you move each mesh, you will see the material update in real-time. The displacement will update once you stop moving the meshes. This method works particularly well for organic rock or stone materials or where you require variety from a limited number of base meshes.
In the example below, only geometry information was captured, so this is a great candidate for triplanar projection.
I baked some masks, such as ambient occlusion and position, and a mask based on normals to use in a moss layer. I‘ve layered some damaged plaster materials from the Library to create a seamless and believable base for the moss to grow on. I then used the Moss Rigid Beard material, tiled it to a detail level that the subdivided geometry could support, and masked it using a baked texture. To add subsurface scattering to the moss material, I enabled Volumetric Scattering in the Transmission shading model on the base layer, which exposed it to the moss layer above. I dialed in some appropriate values for the moss and changed the Mask value to zero to ensure no scattering occurred in the base stone layer. Note: Materials can be dragged from the Library window and dropped on a Material Layer, making it easy to get results quickly.
Similarly, I added a layer of compacted snow to the same statue model using different masks and materials in the following image.
Here is the Lava Aa Cooled material from the Library combined with the Snow Clumps material. It’s very convenient to edit each material in Toolbag without constantly updating textures externally.
In the following scene, a coastal rock formation was captured using photogrammetry, but parts of the model lacked geometric detail, and the albedo texture resolution was low.
Selecting pixels on the low-resolution texture based on color range values made it possible to isolate individual areas such as algae, sand, and base. The areas with low detail were also masked appropriately for each layer, and by using triplanar displacement, detail was salvaged to problem areas by independently tiling multiple layers. These enhancements made it possible to create extreme close-up renders that the original capture alone couldn’t achieve.
Thanks for reading! Start a free, 30-day trial for Toolbag today to create realism in your scene with Displacement materials. For more Toolbag tutorials and quick guides, check out our Resources.
