Skip to content

Shaders in Axmol3

Jump to bottom
halx99 edited this page Jul 12, 2026 · 7 revisions

Shader Migration Guide: GLSL → HLSL 5.1 (WIP)

axmol PR #3233 — Migration of all engine and test shaders from GLSL to HLSL 5.1

Overview

This document describes the complete shader migration from GLSL (OpenGL ES 3.0) to HLSL 5.1 (D3D11/D3D12/Vulkan/GLES) for the axmol engine. The migration covers:

  • 57 builtin engine shaders (axmol/renderer/shaders/)
  • 44 cpp-tests shaders (tests/cpp-tests/Source/shaders/)
  • 2 spine shaders (extensions/spine/shaders/)
  • 19 Live2D shaders (extensions/Live2D/.../shaders/)

Code Structure Changes

New vs Old File Extensions

Old New
.vert _vs.hlsl
.frag _ps.hlsl
.fsh _ps.hlsl
.vsh _vs.hlsl
.vs _vs.hlsl
.fs _ps.hlsl

Stage detection: axslcc now detects shader stage only from file stem suffix (_vs, _ps, _cs). Legacy extension-based detection removed.

AXSLCC.cmake

# Before: supported GLSL + HLSL extensions
axslcc_option(AXSLCC_SOURCE_FILE_EXTENSIONS ".hlsl;.vert;.vsh;.vs;.frag;.fsh;.fs")

# After: only .hlsl
axslcc_option(AXSLCC_SOURCE_FILE_EXTENSIONS ".hlsl")

AXSLCC_VARIANT_DEFINES — Multi-Output Compilation

New in PR #3233: replaces the single-variant AXSLCC_OUTPUT1 property. Supports arbitrary number of compilation variants from one .hlsl source.

set_source_files_properties(
    positionNormalTexture_vs.hlsl
    skinPositionNormalTexture_vs.hlsl
    colorNormalTexture_ps.hlsl
    PROPERTIES AXSLCC_VARIANT_DEFINES "USE_NORMAL_MAPPING=1"
)
# → produces: file, file_1 (with -DUSE_NORMAL_MAPPING=1)

# Multiple variants: semicolons separate variants, commas separate defines within a variant
PROPERTIES AXSLCC_VARIANT_DEFINES "A=1,B=2;C=3"
# → produces: file, file_1 (-DA=1 -DB=2), file_2 (-DC=3)
Property Scope Format
AXSLCC_DEFINES All outputs Comma-separated defines
AXSLCC_VARIANT_DEFINES _N variant outputs only Semicolons separate variants; commas separate defines per variant

Important: variant defines are only applied to the _1, _2, ... outputs. The base output (no suffix) is compiled with only AXSLCC_DEFINES — never with variant defines. This ensures the base output matches the ProgramManager's default program pairing.

Key Files Involved

File Role
axmol/renderer/shaders/base.hlsli Common macros, builtin sampler declarations
axmol/cmake/Modules/AXSLCC.cmake Shader build rules
axslcc/src/ Offline shader compiler

Shader Conversion Patterns

Vertex Shader: GLSL → HLSL

// BEFORE (GLSL)
#version 310 es
layout(location = POSITION) in vec3 a_position;
layout(location = TEXCOORD0) in vec2 a_texCoord;
layout(location = TEXCOORD0) out vec2 v_texCoord;
layout(std140, set = 0, binding = 0) uniform vs_ub {
    mat4 u_MVPMatrix;
};

void main() {
    v_texCoord = a_texCoord;
    gl_Position = u_MVPMatrix * vec4(a_position, 1.0);
}

// AFTER (HLSL)
#include "base.hlsli"

struct VS_IN {
    float3 a_position : POSITION;
    float2 a_texCoord : TEXCOORD0;
};

struct VS_OUT {
    float2 v_texCoord : TEXCOORD0;
    float4 position : SV_Position;
};

cbuffer vs_ub : register(b0, space0) {
    float4x4 u_MVPMatrix;
};

VS_OUT main(VS_IN input) {
    VS_OUT output;
    output.position = mul(u_MVPMatrix, float4(input.a_position, 1.0));
    output.v_texCoord = input.a_texCoord;
    return output;
}

Fragment Shader: GLSL → HLSL

// BEFORE (GLSL)
#version 310 es
precision highp float;
layout(location = TEXCOORD0) in vec2 v_texCoord;
layout(set = 1, binding = 0) uniform sampler2D u_tex0;
layout(std140, set = 0, binding = 1) uniform fs_ub {
    vec4 u_color;
};
layout(location = SV_Target0) out vec4 FragColor;

void main() {
    FragColor = texture(u_tex0, v_texCoord) * u_color;
}

// AFTER (HLSL)
#include "base.hlsli"

struct PS_IN {
    float2 v_texCoord : TEXCOORD0;
};

cbuffer fs_ub : register(b1, space0) {
    float4 u_color;
};

Texture2D u_tex0 : register(t0, space1);

float4 main(PS_IN input) : SV_Target0 {
    return u_tex0.Sample(LinearClamp, input.v_texCoord) * u_color;
}

Shadertoy / Fullscreen PS (no varying inputs)

#include "base.hlsli"

struct PS_IN {
    float4 gl_FragCoord : SV_Position;
};

cbuffer fs_ub : register(b1, space0) {
    float2 center;
    float2 resolution;
    float4 u_Time;
};

float4 main(PS_IN input) : SV_Target0 {
    float2 fragCoord = input.gl_FragCoord.xy;
    // ... shadertoy body ...
}

Type Translation Table

GLSL HLSL
vec2 float2
vec3 float3
vec4 float4
mat4 float4x4
mat3 float3x3
mix(a,b,t) lerp(a,b,t)
fract(x) frac(x)
mod(x,y) fmod(x,y)
texture(sampler, uv) sampler.Sample(LinearClamp, uv)
texture(samplerCube, dir) samplerCube.Sample(LinearClamp, dir)
gl_PointSize = val; output.pointSize = val; (with [[vk::builtin("PointSize")]] annotation)
gl_Position output.position (SV_Position)
gl_FragCoord.xy input.gl_FragCoord.xy (SV_Position)
FragColor = val; return val;
layout(location=SV_Target0) out vec4 FragColor; float4 main(...) : SV_Target0

Semantic Mapping

GLSL Location HLSL Semantic
POSITION (0) POSITION
TEXCOORD0 TEXCOORD0
TEXCOORD1 TEXCOORD1
NORMAL NORMAL
COLOR0 COLOR0
BINORMAL BINORMAL
BLENDINDICES BLENDINDICES
BLENDWEIGHT BLENDWEIGHT

Uniform Buffer Layout

// Vertex shader UBO — set 0 → space 0, binding 0
cbuffer vs_ub : register(b0, space0) { ... }

// Fragment shader UBO — set 0 → space 0, binding 1
cbuffer fs_ub : register(b1, space0) { ... }

Texture and Sampler Bindings

// Texture at t0 — set 1 → space 1
Texture2D u_tex0 : register(t0, space1);
Texture2D u_tex1 : register(t1, space1);
TextureCube u_cubeTex : register(t0, space1);

// Builtin samplers are declared in base.hlsli
// Sampling: use LinearClamp, LinearWrap, PointClamp, etc.
u_tex0.Sample(LinearClamp, uv)

Legacy Cleanup

base.hlsli — Removed Macros

// REMOVED — these were GLES2 legacy
#undef TANGENT
#undef BINORMAL
#undef BLENDINDICES
#undef BLENDWEIGHT
#define TANGENT TEXCOORD6
#define BINORMAL TEXCOORD7
#define BLENDINDICES COLOR1
#define BLENDWEIGHT COLOR2

base.hlsli — Semantic Fix

// BEFORE: SPOTLIGHT overlapped with POINTLIGHT array (TEXCOORD2–5)
#ifndef SPOTLIGHT
#define SPOTLIGHT TEXCOORD4
#endif

// AFTER: moved to TEXCOORD6 (after POINTLIGHT's 4-element range)
#ifndef SPOTLIGHT
#define SPOTLIGHT TEXCOORD6
#endif

Sampler12.h — Deleted

The entire axmol/rhi/d3d12/Sampler12.h file was removed. Its BuiltinSamplers string contained sampler declarations that are now handled by base.hlsli.

colorUtils.hlsli — Matrix Multiply Fix

// BEFORE: DXC rejected implicit float3x3 constructor
return float3x3(colorTransform[0].xyz, colorTransform[1].xyz, colorTransform[2].xyz) * YUV;

// AFTER: explicit matrix construction
float3x3 m = { colorTransform[0].xyz, colorTransform[1].xyz, colorTransform[2].xyz };
return mul(m, YUV);

Tool Changes

axslcc — New Options

Option Description
--dxil Windows only: compile SPIRV-Cross HLSL output to DXIL bytecode via DXC
--dxc-reflect Windows only: use DXC reflection for REFL chunk (stub, validation only)

axslcc — Compilation Pipeline

axslcc performs per-target front-end compilation to enable source-level target macros:

for each target:
  HLSL source + -DAXSLC_TARGET_<LANG>=1
    → glslang → SPIR-V
      → SPIRV-Cross + attribute remap → target output (HLSL/ESSL/GLSL/MSL)
      → SPIR-V raw (for Vulkan, direct copy)
    → (+ --dxbc) SPIRV-Cross HLSL output → FXC (DXBC) or DXC (DXIL)
    → (+ --reflect) REFL chunk from SPIR-V reflection
  → packed into .sc file (axslcc --sc)

axslcc — Key Fixes

Fix File
Attribute remap for HLSL output cross_compiler.cpp
Varying name normalize for GLES cross_compiler.cpp
Save/restore combined sampler names (ESSL/GLSL) cross_compiler.cpp
HLSL source semantic parse for REFL utils.cpp + reflection.cpp
separate_images for HLSL Texture2D reflection.cpp
input. prefix strip from variable names utils.cpp
SEH crash handler + g_inDxcCompile flag main.cpp + compiler.cpp
/EHa for cross-DLL C++ exception catch CMakeLists.txt
N-variant compilation + define parsing refactor AXSLCC.cmake

Backend-Specific Notes

D3D11/D3D12

  • Engine RHI receives HLSL source code from SC files
  • D3D12 driver compiles HLSL → DXIL at runtime using DXC
  • Use --dxil option for offline DXIL pre-compilation

GLES (ANGLE / OpenGL ES 3.0)

  • SPIRV-Cross generates ESSL 300 from SPIR-V
  • Varying names normalized to strip input. prefix
  • GLES linker matches varyings by name

Vulkan

  • SPIR-V binary directly embedded in SC file
  • No cross-compilation needed

Metal (MSL)

  • Not tested yet
  • Metal Y-flip handled by AXSLC_TARGET_MSL macro

Common Pitfalls

PS_IN / VS_OUT Member Declaration Order

Symptom: D3D12 CreateGraphicsPipelineState fails:

D3D12 ERROR: ID3D12Device::CreateGraphicsPipelineState: Vertex Shader - Pixel Shader
linkage error: Signatures between stages are incompatible.
Semantic 'TEXCOORD' of the input stage has a hardware register component mask that
is not a subset of the output of the previous stage.
[STATE_CREATION ERROR #662: CREATEGRAPHICSPIPELINESTATE_SHADER_LINKAGE_REGISTERMASK]

Root Cause: The shader compilation pipeline does not preserve HLSL semantic names for varyings through the intermediate SPIR-V representation:

  1. glslang compiles HLSL to SPIR-V, assigning Location decorations to struct members in declaration order (ignoring the semantic name/index)
  2. SPIRV-Cross cross-compiles SPIR-V to HLSL for D3D targets, generating TEXCOORDn semantics from Location values
  3. If VS_OUT and PS_IN declare members in different orders, the cross-compiled semantics will mismatch

Example — incorrect:

// VS_OUT (from positionTextureColor_vs.hlsl):
struct VS_OUT {
    float4 v_color : COLOR0;       // location 0 → TEXCOORD0
    float2 v_texCoord : TEXCOORD0; // location 1 → TEXCOORD1
    float4 position : SV_Position;
};

// PS_IN (custom):
struct PS_IN {
    float2 v_texCoord : TEXCOORD0; // location 0 → TEXCOORD0 (receives v_color!)
    float4 v_color : COLOR0;       // location 1 → TEXCOORD1 (receives v_texCoord!)
};
// → D3D12: PS expects float4 at TEXCOORD1 but VS writes float2 → PSO rejected

Fix: Ensure PS_IN member declaration order matches VS_OUT:

struct PS_IN {
    float4 v_color : COLOR0;       // location 0 → TEXCOORD0 ✓
    float2 v_texCoord : TEXCOORD0; // location 1 → TEXCOORD1 ✓
};

Affected files: 9 custom PS shaders in tests/cpp-tests/Source/shaders/ paired with positionTextureColor_vs had reversed v_texCoord/v_color order:

File Status
example_Blur_ps.hlsl Fixed
example_Outline_ps.hlsl Fixed
example_Noisy_ps.hlsl Fixed
example_GreyScale_ps.hlsl Fixed
example_Sepia_ps.hlsl Fixed
example_EdgeDetection_ps.hlsl Fixed
example_Bloom_ps.hlsl Fixed
example_CelShading_ps.hlsl Fixed
example_Blur_winrt_ps.hlsl Fixed
example_Normal_ps.hlsl Fixed
example_HorizontalColor_ps.hlsl Fixed
Normal_ps.hlsl Already correct

This issue only affects PS shaders paired with a VS from a different file. When the VS and PS are written together (same prefix), the declaration order is naturally consistent.

VS_IN Member Declaration Order

Symptom: D3D11 debug layer reports at Draw time:

D3D11 ERROR: ID3D11DeviceContext::Draw: Input Assembler - Vertex Shader linkage
error: Signatures between stages are incompatible.
Semantic 'COLOR' is defined for mismatched hardware registers between the output
stage and input stage.
Semantic 'TEXCOORD' is defined for mismatched hardware registers between the output
stage and input stage.

Root Cause: The vertex shader's VS_IN struct declares members in a different order from the VertexLayoutKind layout definition in VertexLayoutManager.cpp. The axslcc pipeline assigns D3D input registers based on VS_IN declaration order. While D3D11 matches IA elements to VS inputs by semantic name+index, some D3D11 debug layers and GPU drivers additionally validate that the "hardware register" indices (derived from declaration order) are consistent between IA output slots and VS input registers.

Example — VertexLayoutKind::DrawNode layout order:

POSITION → TEXCOORD → COLOR

If positionColorTextureAsPointsize_vs.hlsl had:

struct VS_IN {
    float4 a_position : POSITION;
    float4 a_color : COLOR0;        // register v1
    float2 a_texCoord : TEXCOORD0;  // register v2
};

The IA provides TEXCOORD at slot 1 and COLOR at slot 2, but the VS expects COLOR at v1 and TEXCOORD at v2 — register indices swapped.

Fix: Ensure VS_IN declaration order matches the vertex layout:

struct VS_IN {
    float4 a_position : POSITION;
    float2 a_texCoord : TEXCOORD0;  // register v1 → matches IA slot 1
    float4 a_color : COLOR0;        // register v2 → matches IA slot 2
};

Affected builtin engine shader:

File Fix
positionColorTextureAsPointsize_vs.hlsl Reordered COLOR0/TEXCOORD0 to match VertexLayoutKind::DrawNode

Y-Flip Handling with AXSLC_TARGET_* Macros

After migration, the AXSLC_TARGET_* macros were removed from shader sources, but they are still needed because the Y-axis origin differs between platforms (GLSL/ESSL: bottom-left; HLSL/SPIRV/MSL: top-left).

axslcc now performs per-target front-end compilation, passing the appropriate AXSLC_TARGET_<LANG>=1 as a preprocessor define via glslang. This allows #include "base.hlsli" to resolve #ifdef conditionals correctly at source level.

base.hlsli provides two macros:

Macro Purpose
AXSLC_UV_TOP Flag: 0 = bottom-left (GLSL/ESSL), 1 = top-left (HLSL/SPIRV/MSL)
AX_Y_UP(v) Convert Y to bottom-left convention: (AXSLC_UV_TOP ? (1.0 - (v).y) : ((v).y))

vr_vs.hlsl uses AX_Y_UP for Y-up texcoord calculation:

output.v_texCoord = float2(input.a_texCoord.x, AX_Y_UP(input.a_texCoord));

11 fullscreen PS shaders use #if AXSLC_UV_TOP for gl_FragCoord Y-flip:

#if AXSLC_UV_TOP
    float2 fragCoord = float2(input.gl_FragCoord.x, u_screenSize.y - input.gl_FragCoord.y);
#else
    float2 fragCoord = input.gl_FragCoord.xy;
#endif
File Flip source
example_Heart_ps.hlsl u_screenSize.y
example_Julia_ps.hlsl u_screenSize.y
example_Flower_ps.hlsl u_screenSize.y
example_Mandelbrot_ps.hlsl u_screenSize.y
example_Monjori_ps.hlsl u_screenSize.y
example_Twist_ps.hlsl u_screenSize.y
example_LensFlare_ps.hlsl resolution.y
example_Plasma_ps.hlsl u_screenSize.y
shadertoy_FireBall_ps.hlsl u_screenSize.y
shadertoy_Glow_ps.hlsl u_screenSize.y
shadertoy_LensFlare_ps.hlsl u_screenSize.y

Migration Checklist

  • Builtin engine shaders (57 files)
  • cpp-tests shaders (44 GLSL → 50 HLSL)
  • spine shaders (2 files)
  • Live2D shaders (19 files)
  • AXSLCC.cmake: only .hlsl
  • axslcc: only _vs/_ps/_cs stage detection
  • D3D11/D3D12 basic rendering
  • Vulkan basic rendering
  • GLES (ANGLE) basic rendering
  • Metal testing
  • --dxc-reflect DXC reflection extraction
  • solid_circle/solid_capsule SDF shaders (orphan, no test references)

Axmol.dev / Docs / Discord / Reddit

INDEX

GENERAL

MIGRATIONS & UPDATES

FEATURES

BASIC TUTORIALS

ADVANCED TUTORIALS

ABOUT

Clone this wiki locally