Shaders in Axmol3

Shaders in Axmol 3

Axmol 3 uses HLSL as the primary shader authoring language. The engine shader compiler, axslcc, generates the backend shader code and reflection data used by D3D11, D3D12, Vulkan, Metal, OpenGL 3.3, and OpenGL ES 3.x.

This page is written for shader authors and engine users. For low-level compiler and RHI implementation details, see the source tree documentation under docs/hlsl-spec.md and docs/hlsl-faq.md.

What changed in Axmol 3

• New shaders should be written in HLSL 5.1 style.
• Shader stages are identified by filename suffixes such as _vs.hlsl, _ps.hlsl, _fs.hlsl, and _cs.hlsl.
• Vertex inputs are matched by HLSL semantic name and semantic index, not by variable name.
• Textures, samplers, and uniform buffers should use explicit HLSL registers.
• The runtime reflection stores final backend resources: vertex inputs, textures, samplers, uniform buffers, and storage resources where applicable.
• Texture/sampler sampling pairs are compiler-internal lowering data and are not part of the runtime reflection ABI.

File naming

Use these suffixes:

Stage	Suffix	Example
Vertex	_vs.hlsl	positionTexture_vs.hlsl
Pixel / Fragment	_ps.hlsl or _fs.hlsl	positionTexture_ps.hlsl
Compute	_cs.hlsl	particles_cs.hlsl

Use main as the entry point unless a specific build rule says otherwise.

Always include base.hlsli for engine shaders

Most Axmol shaders should start with:

#include "base.hlsli"

base.hlsli provides:

• built-in sampler presets such as LinearClamp, LinearWrap, and PointClamp;
• backend target macros such as AXSLC_TARGET_HLSL, AXSLC_TARGET_SPIRV, AXSLC_TARGET_GLSL, AXSLC_TARGET_ESSL, and AXSLC_TARGET_MSL;
• helper macros used by existing engine shaders, including coordinate helpers.

Vertex input semantics

Axmol does not bind vertex data by HLSL member name. It binds by semantic base name and semantic index.

struct VS_IN
{
    float3 position : POSITION;
    float2 uv       : TEXCOORD0;
    float4 color    : COLOR0;
};

POSITION is equivalent to POSITION0. TEXCOORD1 means semantic base TEXCOORD with index 1.

Common mesh semantics:

Semantic	Typical use
POSITION	vertex position
COLOR0, COLOR1	vertex colors; COLOR1 is used by some two-color pipelines
TEXCOORD0 - TEXCOORD3	UV channels
NORMAL	normal
TANGENT	tangent
BINORMAL	binormal
BLENDWEIGHT	skinning weights
BLENDINDICES	skinning joint indices

Custom rendering code may use custom semantics:

struct VS_IN
{
    float2 localPosition : LOCAL_POSITION0;
    float4 instanceColor : INSTANCE_COLOR0;
};

The C++ side must query the same semantic base and index. Avoid designing code that depends on member names such as a_position or a_texCoord.

Vertex output and pixel input

Use matching semantics between vertex shader outputs and pixel shader inputs. Variable names may differ; semantics must match.

struct VS_OUT
{
    float4 position : SV_Position;
    float2 uv       : TEXCOORD0;
    float4 color    : COLOR0;
};

struct PS_IN
{
    float2 texcoord : TEXCOORD0;
    float4 tint     : COLOR0;
};

SV_Position is special:

• vertex shaders write clip-space position through SV_Position;
• pixel shaders may read screen-space position through SV_Position;
• it is not a normal user varying like TEXCOORD0.

Dynamic Point Size Output

Dynamic point size output is backend dependent. Axmol HLSL supports dynamic point size output on OpenGL/OpenGL ES and Vulkan. To declare a dynamic point size output, use:

struct VS_OUT
{
    float4 position : SV_Position;
    [[vk::builtin("PointSize")]] float pointSize : PSIZE;
};

The vk::builtin("PointSize") attribute is recognized by axslcc and enables Vulkan point size output. OpenGL/OpenGL ES backends translate this output to gl_PointSize. D3D11, D3D12, and Metal backends currently render points with a fixed size of 1 pixel.

Uniform buffers

Use explicit constant buffer registers.

cbuffer vs_ub : register(b0, space0)
{
    float4x4 u_MVPMatrix;
};

cbuffer fs_ub : register(b1, space0)
{
    float4 u_color;
};

Current engine convention:

Register	Use
b0, space0	vertex-stage uniforms
b1, space0	pixel-stage uniforms

Keep uniform names stable when engine code looks them up by name.

Textures

Declare textures in space1 with t# registers.

Texture2D u_tex0 : register(t0, space1);
Texture2D u_tex1 : register(t1, space1);
TextureCube u_Env : register(t0, space1);

Texture binding and sampler binding are independent in HLSL:

Texture2D albedo : register(t3, space1);
float4 c = albedo.Sample(LinearClamp, uv); // texture t3, sampler s0

Do not assume the sampler slot equals the texture slot.

Samplers

Axmol currently supports two practical sampler sources:

Source	Meaning	Recommended use
Shader preset	sampler state comes from base.hlsli presets	preferred for most shaders
TextureOwned	sampler state comes from exactly one associated texture/resource	one non-preset sampler per owner texture

There is no public runtime API for binding arbitrary custom sampler objects by shader sampler name. If you need predictable cross-backend behavior, use the built-in presets.

Shader preset samplers

Use sampler presets from base.hlsli:

Texture2D u_tex0 : register(t0, space1);

float4 main(PS_IN input) : SV_Target0
{
    return u_tex0.Sample(LinearClamp, input.uv);
}

Common presets include:

Name	Register	Kind
LinearClamp	s0, space1	linear filtering, clamp
LinearWrap	s1, space1	linear filtering, wrap
PointClamp	s4, space1	point filtering, clamp
PointWrap	s5, space1	point filtering, wrap
AnisoClamp	s12, space1	anisotropic filtering, clamp
AnisoWrap	s13, space1	anisotropic filtering, wrap
ShadowCmpClamp	s16, space1	comparison sampler, clamp
LinearNoMipClamp	s20, space1	linear filtering, no mip, clamp
PointNoMipClamp	s21, space1	point filtering, no mip, clamp

See base.hlsli for the complete preset list.

TextureOwned samplers

TextureOwned sampling means the sampler state is taken from one associated texture resource bound by the engine. The texture may internally use or reuse an existing built-in sampler state; the shader register still follows Axmol's sampler ABI.

The texture register and sampler register do not need to have the same number, but s0 - s21 are reserved binding slots for ShaderPreset samplers declared in base.hlsli. TextureOwned samplers should use s22 or higher:

Texture2D u_albedo : register(t3, space1);
SamplerState u_albedoSampler : register(s22, space1);

float4 c = u_albedo.Sample(u_albedoSampler, uv);

This is not an arbitrary custom sampler. It is a request to use the sampler state owned by texture binding t3, then bind it at sampler binding s22.

A TextureOwned sampler must have exactly one owner texture. If multiple textures should share the same sampler state, use a ShaderPreset such as LinearClamp.

OpenGL and OpenGL ES sampler behavior

D3D11, D3D12, Vulkan separate mode, and Metal can represent separate textures and samplers directly.

OpenGL 3.3 and OpenGL ES 3.x expose combined sampler uniforms in GLSL/ESSL, such as sampler2D. axslcc lowers HLSL texture + sampler usage into final combined GLSL resources at compile time. The runtime reflection for GL/GLES describes those final combined uniforms.

For cross-backend shaders, avoid sampling one texture with multiple sampler states:

// Avoid for portable Axmol shaders:
float4 a = u_tex0.Sample(LinearClamp, uv);
float4 b = u_tex0.Sample(PointClamp, uv);

Use separate texture bindings or choose one sampler state for that texture.

Texture sampling operations

Common GLSL-to-HLSL mappings:

GLSL	HLSL
texture(tex, uv)	tex.Sample(LinearClamp, uv)
textureLod(tex, uv, lod)	tex.SampleLevel(LinearClamp, uv, lod)
textureGrad(tex, uv, dx, dy)	tex.SampleGrad(LinearClamp, uv, dx, dy)
textureProj(tex, coord)	write the projection explicitly, then sample

When migrating from GLSL, verify the exact sampler, coordinate, LOD, gradient, and channel usage. AI-assisted translations are useful starting points, but they can easily change material behavior.

UVs, Y flip, and AX_Y_UP

Do not add Y flips just because the shader is HLSL. Decide from the meaning of the data.

Coordinate meaning	Recommended handling
Mesh or asset UV is already in the texture's expected convention	pass it through unchanged
Asset data or the original shader intentionally used the opposite UV convention	explicitly convert the data, for example uv.y = 1.0 - uv.y
Screen-space or render-target coordinate needs a target-origin conversion	use AX_Y_UP(coord)

The short rule:

• 1.0 - uv.y means “this input data should be inverted.”
• AX_Y_UP(coord) means “convert this backend-dependent coordinate to Axmol's expected Y-up convention.”

Some shaders need a manual data-space flip. Some do not. Fullscreen effects, render-target reads, and screen-coordinate math are the cases most likely to need AX_Y_UP.

Color space and channel checks

When converting GLSL to HLSL, pay special attention to:

• mask textures using .r, .g, .b, or .a;
• normal maps and tangent-space reconstruction;
• specular, metallic, roughness, and occlusion channels;
• sRGB versus linear sampling assumptions;
• alpha premultiplication;
• branches controlled by material defines or uniform flags.

If a migrated material renders too dark or has unexpected black regions, first compare the original GLSL, the HLSL source, and the generated GLSL/ESSL before changing RHI binding code.

Target macros

axslcc compiles backend variants with target macros:

Backend output	Defines
D3D/HLSL	AXSLC_TARGET_HLSL=1
Vulkan/SPIR-V	AXSLC_TARGET_SPIRV=1
Metal/MSL	AXSLC_TARGET_MSL=1
OpenGL/GLSL	AXSLC_TARGET_GLSL=1
OpenGL ES/ESSL	AXSLC_TARGET_ESSL=1, AXSLC_TARGET_GLSL=1

Use backend conditionals sparingly. Prefer one source path unless the coordinate origin, platform feature, or backend language really requires a difference.

Build and shader recompilation

Axmol's CMake rules compile .hlsl sources into .sc shader packages through axslcc.

During Axmol 3 alpha, reflection layout and shader ABI may still change. After updating axslcc or changing reflection-related code, rebuild shader packages with the matching compiler before testing runtime rendering.

Do not mix old .sc files with a newer runtime reflection reader.

Migration checklist

• Use HLSL files with _vs.hlsl, _ps.hlsl, _fs.hlsl, or _cs.hlsl suffixes.
• Include base.hlsli when using Axmol sampler presets or helper macros.
• Bind vertex data by semantic name and index, not by variable name.
• Use explicit registers: bN, space0 for uniform buffers, tN, space1 for textures, s0 - s21 for ShaderPreset samplers from base.hlsli, and s22+ for TextureOwned samplers.
• Prefer built-in sampler presets such as LinearClamp.
• Do not assume tN == sN; texture and sampler bindings are independent.
• Avoid one texture sampled with multiple sampler states if GL/GLES is a target.
• Treat UV flips as data/coordinate semantics, not as a mechanical HLSL rule.
• Validate texture channels and color-space assumptions when migrating materials.
• Recompile shaders after changing axslcc or reflection layout.