By Jeff Russell
Toolbag 3 provides a framework for users to customize the material shader system. This can be a powerful tool for users to extend Toolbag’s rendering capabilities, and for these improvements to be shared with others. The purpose of this article is to provide enough of an introduction to get a developer or technical artist started. Some basic knowledge of shader development and languages will be helpful; users looking to install custom shaders should see our Toolbag Add-On Library instead.
For those of you who might prefer a video introduction, our friend Chris Perrella has prepared a tutorial demonstrating the basics of custom shader creation and use. We suggest you give it a look before reading further:
Location
Toolbag’s material shaders are located in the installation directory, inside data/shader/mat. User shaders belong in data/shader/mat/custom; any files placed there will appear in the shader dropdown in the “Custom” material module as the tutorial video illustrates. It’s worth exploring the contents of the mat directory as it contains source for the built-in shader modules in Toolbag, all of which are written in the same language and structure available to user shaders. Of particular note are mat.frag and state.frag, which will be discussed in more detail below. There are also two example shaders included with Toolbag in the custom directory, and these provide good examples of common shading operations in Toolbag.
Please note also that Toolbag’s console window (accessible through the Help -> Dev menu, or by pressing Ctrl + ~ on Windows and Cmd + Shift + C on Mac) will display any syntax errors during shader load or reload. This can be very helpful for development and debugging.
Language
Marmoset Toolbag uses a somewhat customized shader language, which is a kind of union of HLSL and GLSL syntax conventions. In many cases, both sets of keywords work (e.g. both vec3 and float3 are valid), and in other cases custom macros are required. This syntax allows Toolbag shaders to compile and run in Direct3D, OpenGL, and more recently Apple’s Metal. If you follow these conventions, your shader should work correctly on a wide variety of operating systems and hardware.
One common example of this unique syntax, which may seem unusual to those of you used to HLSL, relates to textures. Textures are declared with the USE_TEXTURE2D macro, and are accessed with GLSL-like commands. Here is a simple example of texture usage in a Toolbag shader:
A full description of the shading language is beyond the scope of this article. Reading through other files for examples is likely to be the most productive route to learning the ins and outs. Some basic reference for GLSL may also be helpful as Toolbag’s shading language most closely resembles this.
Subroutines
Marmoset material shaders are organized as a collection of subroutines for various predefined ‘slots’, all of which are called from the main shader file, mat.frag. Custom shaders may extend or replace any of these subroutines, and by doing so significantly alter the rendering behavior of the material. For example, a custom shader my supply a new “Surface” subroutine to alter the standard normal mapping functions. The following shader extends the Surface subroutine to adjust the normal to point in a different direction. This will affect all subsequent shading processing, including diffuse and specular lighting:
Other material subroutines in Toolbag can be replaced and extended in similar fashion. Here is a full listing of the available subroutines, in the order in which they run:
| Subroutine | Description |
| Premerge | Preparatory work at the start of a shader. |
| Surface | Specifies the material surface; typically a normal mapping function. |
| Microsurface | Sets gloss/roughness for shading. |
| Albedo | Sets diffuse albedo/color. |
| Reflectivity | Sets specular reflectivity. |
| DiffusionEnv | Computes diffuse environment lighting. |
| Diffusion | Computes diffuse lighting (called once per light). |
| ReflectionEnv | Computes specular environment lighting. |
| Reflection | Computes specular lighting (called once per light). |
| Occlusion | Applies occlusion, typically ambient occlusion (called after lighting). |
| Emissive | Emissive light, typically glow. |
| Transparency | Any transparency work happens here. |
| Merge | Performs the final merge of above values into the render color outputs. |
All of these subroutines take a FragmentState structure by reference (hence the inout tag). This structure holds all intermediate values used by a Marmoset material shader, and is an excellent place to start reading. This structure is defined in state.frag, and several of its key members are listed here for reference:
| Type | Name | Purpose |
| vec3 | vertexPosition | Position in 3D space. |
| vec3 | vertexEye | Unit vector from the position to the camera. |
| float | vertexEyeDistance | Direct distance from the position to the camera. |
| vec2 | vertexTexCoord | Mesh texture coordinates. |
| vec2 | vertexTexCoordSecondary | Secondary mesh texture coordinates; often absent/unused. |
| vec4 | vertexColor | Mesh color; will be white if absent. |
| vec3 | vertexNormal | Mesh normal vector. |
| vec3 | vertexTangent | Mesh tangent vector. |
| vec3 | vertexBitangent | Mesh bitangent vector. |
| vec2 | screenTexCoord | Coordinates for screen position in [0,1]. A full-screen texture can be sampled with this. |
| float | screenDepth | Post-projection depth value. Not the same as vertexPosition.z. |
| uint | sampleCoverage | A bit mask for sample coverage; typically only used for voxelization pass. |
| int | instanceID | Instance number; typically only used for voxelization pass. |
| vec4 | albedo | Albedo or diffuse color in rgb, opacity optionally in alpha. |
| vec3 | normal | Surface normal direction as a unit vector. |
| float | gloss | Surface roughness as a scalar on [0,1]. |
| vec3 | reflectivity | Colored specular reflectivity. |
| vec3 | fresnel | Fresnel strength. |
| vec3 | diffuseLight | Sum of all diffuse lighting. |
| vec3 | specularLight | Sum of all specular lighting. |
| vec3 | emissiveLight | Sum of all emissive lighting. |
| vec4 | generic0…3 | Four generic values for special use by some subroutines. |
| vec4 | diffuseGI | Global diffuse illumination in RGB, mask in A. Not always present. |
| vec4 | specularGI | Global specular illumination in RGB, mask in A. Not always present. |
| vec4 | output0…7 | Final render color outputs. Usually only modified in the Merge subroutine. |
All of these subroutines take a FragmentState structure by reference (hence the inout tag). This structure holds all intermediate values used by a Marmoset material shader, and is an excellent place to start reading. This structure is defined in state.frag, and several of its key members are listed here for reference:
Some lighting functions, like Diffusion and Reflection, also receive a LightParams structure describing the current light being rendered. For a full definition of this parameter, read other/lightParams.frag.
Parameters
Most custom shaders will need to pass constant parameters to control their operation, and most users will want to see these values exposed with a graphical interface. Fortunately, Toolbag provides exactly such a mechanism. Toolbag will automatically parse your custom shader code looking for ‘uniform’ variables declared in global scope, and create UI based on their type. All float, integer, vector, and scalar types are supported, along with textures. Shader authors may supply comments to give additional information about how the interface should be constructed. A quick example:
In this snippet, two float scalars are declared as uniform parameters, and both will appear in the UI. The difference is that the second one, tagged with formatters, will be displayed as “My Parameter” instead of just “uVal2”, will have a slider ranging from 0 to pi instead of 0 to 1, and will be given a default value of 2.0. Decorating uniform parameters with these formatting options makes for friendlier interfaces to custom shaders. A full list of formatting options follows:
| Directive | Options | Description |
| name | “Name” | Uses “Name” as the display name. |
| default | v0 v1 … | A default value for numerical parameters. May be a vector or single value. |
| min | v0 v1 … | A minimum value for numerical parameters. May be a vector or single value. |
| max | v0 v1 … | A maximum value for numerical parameters. May be a vector or single value. |
| color | – | Displays 3D vectors as color pickers instead of text boxes. |
| bool | – | This option presents integer parameters as checkboxes instead of sliders. |
| srgb | – | Sets a texture to use the sRGB color space where possible. |
| labels | “Name0”, “Name1”, … | Displays an enumerated list of options for integer parameters instead of a slider. Must be the last element in the comment line. |
Passes
Toolbag performs a number of passes while rendering, depending on settings and the needs of the final frame. Each pass involves drawing some or all of the scene geometry with different shader options. Material subroutines are not always needed in every pass, or are sometimes only appropriate for certain passes. If you notice your custom shader is interfering with effects like GI, shadows, or others, you may want to adjust the passes with which your shader interacts (custom shader code is run in all passes by default). This can be done by checking #define values. For example, let’s say you wanted some code to only apply to the lighting pass (and not, for example, shadowing, wireframe, or other passes):
A full listing of pass defines follows:
| Pass | Description |
| MATERIAL_PASS_PREPASS | Geometry pre-pass; fills position & normal buffer before main render. |
| MATERIAL_PASS_LIGHT | Main lighting pass, includes light from skybox and light sources. |
| MATERIAL_PASS_VOXELIZATION | Voxelization/lighting for GI. Only runs when GI is enabled. |
| MATERIAL_PASS_WIREFRAME | Wireframe pass; not usually of interest. |
| MATERIAL_PASS_SHADOWMAP | Shadow map pass, runs when meshes are rendered into shadow textures. |
| MATERIAL_PASS_EXPORT | Marmoset Viewer or other texture export pass (not present in renders). |
| MATERIAL_PASS_SCATTER | Subsurface scatter prepass; writes values critical to scatter effects. |
Vertex & Tessellation
In addition to custom fragment shader routines, it is also possible to write shaders for the vertex shader (.vert), as well as hull and domain tessellation shaders (.hull, .dom). The process for these stages is somewhat similar, however there are some obvious differences in the state structure and available parameters. If you want to go beyond altering shading and adjust the geometry itself, you will need to write a vertex or tessellation shader module. This can be done by adding a new file with a proper extension but the same name as your .frag file. Toolbag will load multiple files with matching names to their appropriate stages, which would allow a .frag and .vert pair to be distributed together, for example. The user may select either in the custom shader UI and both will be loaded.
Questions?
Shader development is always an involved task. Patience, persistence, and curiosity are rewarded in this endeavor. But if you’ve been poking around and reading example code and you’re still stuck on something, you might try the Toolbag User Group on Facebook. There are several knowledgeable users there including shader developers and some of our own staff. Come and say hello, and show us anything cool you’ve made. Happy shading!
