Load ShaderConductor dependencies on Mac and Linux

[Jorgemagic]

The dynamic libraries on Mac using rpath mechanisms to load dependency libraries, so is important to be in mind when you create a dylib, you must specify correct ID, dependencies and rpaths where linker will find dependency libraries.

If you download ShaderConductor Mac dylib from AzurePipeline artifacts and you use for example MacDependencies app, you can see:

File: libShaderConductor.dylib
ID: @rpath/libShaderConductor.dylib
Dependency with: @rpath/libdxcompiler.3.7.dylib
Rpaths: /Users/vts/agent/2.155.1/work/1/s/Build/ninja-osx-clang9-x64-Release/External/DirectXShaderCompiler/lib.

I created a netcore 3.0 app and make a NativeLibrary.Load to load libShaderConductor.dylib but it produces a fail when load dxcompiler library dependency.

Issues:
Dependency filename is wrong, @rpath/libdxcompiler.3.7.dylib must be @rpath/libdxcompiler.dylib
RPaths is wrong, RPaths could be @loader_path; @executable_path

Is possible fix this references using install_name_tool on Mac but maybe is possible fix it from ninja script.

This is an interesting article about this issue on Mac.

Edit. In Linux exists the same issue, it is possible fix with patchelf command but would be fixed in ninja script.

patchelf --set-sorname libdxcompiler.so libdxcompiler.so
patchelf --set-rpath '$ORIGIN' libShaderConductor.so

[JoelOtter]

It'd be good to get this fixed if possible, as it's blocking our adoption of this software on macOS. :) I'm happy to try and have a crack at it myself if you're able to direct me towards where the linking gets set up in the build process!

To anyone looking at this - it's possible to get it to work by renaming libdxcompiler.dylib to libdxcompiler.3.7.dylib and then symlinking the original filename to the new one (not sure if linker can resolve symlinks but dlopen can)

[gongminmin]

I'm not farmilar with the shared lib on either linux or macos. The CI runs tests on these systems but doesn't fail. Maybe because they are resolved by linker, not dlopen?

Rename and symlink sound a good way. Does anyone can provide a PR? Thanks.

[trojanfoe]

It took me about 30 minutes with otool, install_name_tool and codesign to get the binary running on macOS 10.15.

I had to change the rpath for both the binary and the .dylibs. I had to change the name of the dependencies on libdxcompiler.3.7.dylib and I had to create an .entitlements file in order to turn off hardened runtime and codesign everything with my Apple Developer certificate so it would load the dylibs.

Even then it would not work until I did sudo spctl --master-disable :|

[gongminmin]

Get back to this issue after all these years. I had a fix that might help. But I don't have en environment to test it. Anyone like to have a try? The change is in: gongminmin@05f4536