Shaders in Axmol3

Shaders in Axmol 3

Axmol 3 uses HLSL as the primary shader authoring language. The engine shader compiler, axslcc, generates the backend shader code and reflection data used by D3D11, D3D12, Vulkan, Metal, OpenGL 3.3, and OpenGL ES 3.x.

This page is written for shader authors and engine users. For low-level compiler and RHI implementation details, see the source tree documentation under docs/hlsl-spec.md and docs/hlsl-faq.md.

What changed in Axmol 3

• New shaders should use HLSL 2021 syntax within Axmol's Shader Model 5.1-compatible portable subset.
• Shader stages are identified by filename suffixes such as _vs.hlsl, _ps.hlsl, _fs.hlsl, and _cs.hlsl.
• Vertex inputs are matched by HLSL semantic name and semantic index, not by variable name.
• Textures, samplers, and uniform buffers should be declared without manual HLSL register(...) annotations; axslcc assigns deterministic bindings.
• The runtime reflection stores final backend resources: vertex inputs, textures, samplers, uniform buffers, and storage resources where applicable.
• Texture/sampler sampling pairs are compiler-internal lowering data and are not part of the runtime reflection ABI.

File naming

Use these suffixes:

Stage	Suffix	Example
Vertex	_vs.hlsl	positionTexture_vs.hlsl
Pixel / Fragment	_ps.hlsl or _fs.hlsl	positionTexture_ps.hlsl
Compute	_cs.hlsl	particles_cs.hlsl

Use main as the entry point unless a specific build rule says otherwise.

Always include base.hlsli for engine shaders

Most Axmol shaders should start with:

#include "base.hlsli"

base.hlsli provides:

• built-in sampler presets such as LinearClamp, LinearWrap, and PointClamp;
• backend target macros such as AXSLC_TARGET_HLSL, AXSLC_TARGET_SPIRV, AXSLC_TARGET_GLSL, AXSLC_TARGET_ESSL, and AXSLC_TARGET_MSL;
• helper macros used by existing engine shaders, including coordinate helpers.

Vertex input semantics

Axmol does not bind vertex data by HLSL member name. It binds by semantic base name and semantic index.

struct VS_IN
{
    float3 position : POSITION;
    float2 uv       : TEXCOORD0;
    float4 color    : COLOR0;
};

POSITION is equivalent to POSITION0. TEXCOORD1 means semantic base TEXCOORD with index 1.

Common mesh semantics:

Semantic	Typical use
POSITION	vertex position
COLOR0, COLOR1	vertex colors; COLOR1 is used by some two-color pipelines
TEXCOORD0 - TEXCOORD3	UV channels
NORMAL	normal
TANGENT	tangent
BINORMAL	binormal
BLENDWEIGHT	skinning weights
BLENDINDICES	skinning joint indices

Custom rendering code may use custom semantics:

struct VS_IN
{
    float2 localPosition : LOCAL_POSITION0;
    float4 instanceColor : INSTANCE_COLOR0;
};

The C++ side must query the same semantic base and index. Avoid designing code that depends on member names such as a_position or a_texCoord.

Vertex output and pixel input

Use matching semantics between vertex shader outputs and pixel shader inputs. Variable names may differ; semantics must match.

struct VS_OUT
{
    float4 position : SV_Position;
    float2 uv       : TEXCOORD0;
    float4 color    : COLOR0;
};

struct PS_IN
{
    float2 texcoord : TEXCOORD0;
    float4 tint     : COLOR0;
};

SV_Position is special:

• vertex shaders write clip-space position through SV_Position;
• pixel shaders may read screen-space position through SV_Position;
• it is not a normal user varying like TEXCOORD0.

Dynamic Point Size Output

Dynamic point size output is backend dependent. Axmol HLSL supports dynamic point size output on OpenGL/OpenGL ES and Vulkan. To declare a dynamic point size output, use:

struct VS_OUT
{
    float4 position : SV_Position;
    [[vk::builtin("PointSize")]] float pointSize : PSIZE;
};

The [[vk::builtin("PointSize")]] attribute is recognized by axslcc and enables Vulkan point size output. OpenGL/OpenGL ES backends translate this output to gl_PointSize. D3D11, D3D12, and Metal backends currently render points with a fixed size of 1 pixel.

Uniform buffers

Declare constant buffers without explicit registers.

cbuffer vs_ub
{
    float4x4 u_MVPMatrix;
};

cbuffer fs_ub
{
    float4 u_color;
};

Current internal engine convention:

Assignment	Use
b0, space0	vertex-stage uniforms
b1, space0	pixel-stage uniforms

These assignments are reflection/runtime details. User shaders should not spell them manually.

Keep uniform names stable when engine code looks them up by name.

Textures

Declare textures without explicit registers.

Texture2D u_tex0;
Texture2D u_tex1;
TextureCube u_Env;

axslcc assigns textures to deterministic backend bindings in source order. Texture binding and sampler binding are independent in HLSL:

Texture2D albedo;
float4 c = albedo.Sample(LinearClamp, uv);

Do not assume the sampler slot equals the texture slot.

Samplers

Axmol supports two practical sampler sources:

Source	Meaning	Recommended use
Shader preset	sampler state comes from base.hlsli presets	preferred for most shaders
Custom sampler	sampler state is registered by name from C++	use when presets do not fit

Preset samplers are fixed engine-wide. Custom samplers are Program-local and must be registered through SamplerRegistry before creating the Program.

Shader preset samplers

Use sampler presets from base.hlsli:

Texture2D u_tex0;

float4 main(PS_IN input) : SV_Target0
{
    return u_tex0.Sample(LinearClamp, input.uv);
}

Common presets include:

Name	Internal binding	Kind
LinearClamp	s0, space1	linear filtering, clamp
LinearWrap	s1, space1	linear filtering, wrap
PointClamp	s4, space1	point filtering, clamp
PointWrap	s5, space1	point filtering, wrap
AnisoClamp	s12, space1	anisotropic filtering, clamp
AnisoWrap	s13, space1	anisotropic filtering, wrap
ShadowCmpClamp	s16, space1	comparison sampler, clamp
LinearNoMipClamp	s20, space1	linear filtering, no mip, clamp
PointNoMipClamp	s21, space1	point filtering, no mip, clamp

See base.hlsli for the complete preset list.

Custom samplers

If the built-in presets do not fit, declare your own SamplerState:

Texture2D u_tex0;
SamplerState mySampler;

float4 main(PS_IN input) : SV_Target0
{
    return u_tex0.Sample(mySampler, input.uv);
}

Register the sampler before loading the shader Program:

rhi::SamplerDesc desc;
desc.minFilter    = rhi::SamplerFilter::MIN_LINEAR;
desc.magFilter    = rhi::SamplerFilter::MAG_LINEAR;
desc.sAddressMode = rhi::SamplerAddressMode::REPEAT;
desc.tAddressMode = rhi::SamplerAddressMode::REPEAT;
SamplerRegistry::getInstance()->registerSampler("mySampler", desc);

Rules:

• Custom sampler names must be unique within the shader.
• Custom sampler arrays are not supported.
• Multiple custom samplers are supported in one shader.
• Registration order does not affect shader logical bindings.

OpenGL and OpenGL ES sampler behavior

D3D11, D3D12, Vulkan separate mode, and Metal can represent separate textures and samplers directly.

OpenGL 3.3 and OpenGL ES 3.x expose combined sampler uniforms in GLSL/ESSL, such as sampler2D. axslcc lowers HLSL texture + sampler usage into final combined GLSL resources at compile time. The runtime reflection for GL/GLES describes those final combined uniforms.

For cross-backend shaders, avoid sampling one texture with multiple sampler states:

// Avoid for portable Axmol shaders:
float4 a = u_tex0.Sample(LinearClamp, uv);
float4 b = u_tex0.Sample(PointClamp, uv);

Use separate texture bindings or choose one sampler state for that texture.

Multiple custom samplers are portable when each texture binding uses only one sampler state:

Texture2D u_clampedTex;
Texture2D u_wrappedTex;
SamplerState clampSampler;
SamplerState wrapSampler;

float4 a = u_clampedTex.Sample(clampSampler, uv);
float4 b = u_wrappedTex.Sample(wrapSampler, uv);

Texture sampling operations

Common GLSL-to-HLSL mappings:

GLSL	HLSL
texture(tex, uv)	tex.Sample(LinearClamp, uv)
textureLod(tex, uv, lod)	tex.SampleLevel(LinearClamp, uv, lod)
textureGrad(tex, uv, dx, dy)	tex.SampleGrad(LinearClamp, uv, dx, dy)
textureProj(tex, coord)	write the projection explicitly, then sample

When migrating from GLSL, verify the exact sampler, coordinate, LOD, gradient, and channel usage. AI-assisted translations are useful starting points, but they can easily change material behavior.

Unsupported HLSL resource forms

Some legal HLSL resource forms are intentionally rejected until Axmol has a defined portable RHI binding model for them. Examples include:

• ConstantBuffer<T>; use ordinary cbuffer declarations.
• Texture2DMS and Texture2DMSArray.
• AppendStructuredBuffer<T> and ConsumeStructuredBuffer<T>.
• rasterizer ordered resources such as RasterizerOrderedTexture2D<T>.

UVs, Y flip, and AX_Y_UP

Do not add Y flips just because the shader is HLSL. Decide from the meaning of the data.

Coordinate meaning	Recommended handling
Mesh or asset UV is already in the texture's expected convention	pass it through unchanged
Asset data or the original shader intentionally used the opposite UV convention	explicitly convert the data, for example uv.y = 1.0 - uv.y
Screen-space or render-target coordinate needs a target-origin conversion	use AX_Y_UP(coord)

The short rule:

• 1.0 - uv.y means “this input data should be inverted.”
• AX_Y_UP(coord) means “convert this backend-dependent coordinate to Axmol's expected Y-up convention.”

Some shaders need a manual data-space flip. Some do not. Fullscreen effects, render-target reads, and screen-coordinate math are the cases most likely to need AX_Y_UP.

Color space and channel checks

When converting GLSL to HLSL, pay special attention to:

• mask textures using .r, .g, .b, or .a;
• normal maps and tangent-space reconstruction;
• specular, metallic, roughness, and occlusion channels;
• sRGB versus linear sampling assumptions;
• alpha premultiplication;
• branches controlled by material defines or uniform flags.

If a migrated material renders too dark or has unexpected black regions, first compare the original GLSL, the HLSL source, and the generated GLSL/ESSL before changing RHI binding code.

Target macros

axslcc compiles backend variants with target macros:

Backend output	Defines
D3D/HLSL	AXSLC_TARGET_HLSL=1
Vulkan/SPIR-V	AXSLC_TARGET_SPIRV=1
Metal/MSL	AXSLC_TARGET_MSL=1
OpenGL/GLSL	AXSLC_TARGET_GLSL=1
OpenGL ES/ESSL	AXSLC_TARGET_ESSL=1, AXSLC_TARGET_GLSL=1

Use backend conditionals sparingly. Prefer one source path unless the coordinate origin, platform feature, or backend language really requires a difference.

Build and shader recompilation

Axmol's CMake rules compile .hlsl sources into .sc shader packages through axslcc.

During Axmol 3 alpha, reflection layout and shader ABI may still change. After updating axslcc or changing reflection-related code, rebuild shader packages with the matching compiler before testing runtime rendering.

Do not mix old .sc files with a newer runtime reflection reader.

Migration checklist

• Use HLSL files with _vs.hlsl, _ps.hlsl, _fs.hlsl, or _cs.hlsl suffixes.
• Include base.hlsli when using Axmol sampler presets or helper macros.
• Bind vertex data by semantic name and index, not by variable name.
• Declare resources without manual register(...) annotations; axslcc assigns backend bindings.
• Prefer built-in sampler presets such as LinearClamp.
• Use SamplerRegistry for custom sampler states, and register custom sampler names before creating the Program.
• Do not assume tN == sN; texture and sampler bindings are independent.
• Avoid one texture sampled with multiple sampler states if GL/GLES is a target.
• Treat UV flips as data/coordinate semantics, not as a mechanical HLSL rule.
• Validate texture channels and color-space assumptions when migrating materials.
• Recompile shaders after changing axslcc or reflection layout.