Consider adding Windows on ARM build of llama.dll to LLamaSharp.Backend.Cpu

[david-nordvall]

The llama.cpp project added a native binary build of llama.dll for the Windows on ARM platform in release b2399 (with the rather misleading filename llama-b2399-bin-win-arm64-x64.zip). Please consider adding this to the LlamaSharp.Backend.Cpu NuGet package. 🙂

[AsakusaRinne]

Hi, we're certainly willing to add such a backend. Could you please test whether it works well with current LLamaSharp? Though we could compile and test it with github actions, the period of debugging it will be much longer since we don't have arm windows device. So if you'd like to help it'll save a lot of time. :)

Here're some steps for it:

1. Clone llama.cpp and checkout to 3ab8b3a92ede46df88bc5a2dfca3777de4a2b2b6.
2. Compile it on your PC and get a file llama.dll.
3. Modify the code here to use NativeLibraryConfig.Instance.WithLibrary to load your compiled library file.
4. Run the examples one by one and see if they all works well.

If all things above go well, what we need to do is just adding it to CI and release next version. :)

[martindevans]

Unfortunately it looks like Github Actions doesn't have Windows+ARM available (docs) :(

Edit: Note that's not a total blocker for this. I think we could cross compile the DLLs from Windows x64 Actions, it just means we couldn't test it.

[david-nordvall]

And using the ARM DLL from llama.cpp's official releases isn't a viable workaround?

[AsakusaRinne]

And using the ARM DLL from llama.cpp's official releases isn't a viable workaround?

Using dlls from llama.cpp's release will bring some troubles for auto binary-updating in CI, but I think we could make it as long as you've confirmed that LLamaSharp works well with that dll.

[HillPhelmuth]

I have a Windows Arm device. The latest llama.cpp releases have two options for Windows arm64: llama-b3510-bin-win-llvm-arm64.zip and llama-b3510-bin-win-msvc-arm64.zip

llama-b3510-bin-win-llvm-arm64.zip: Build has llama.dll, which when loaded using NativeLibraryConfig.All.WithLibrary in Llama.Examples loads and runs the examples with no issues.

llama-b3510-bin-win-msvc-arm64.zip: Build has the usual libllama.dll and when I attempt to load this using NativeLibraryConfig.All.WithLibrary I get the same LLama.Exceptions.RuntimeError: The native library cannot be correctly loaded. It could be one of the following reasons: error as I do with the current native libraries in LLamaSharp.Backend.Cpu.

I can't really say why the llvm build works fine but the msvc build fails to load. The differences between various c++ compilers/toolsets is a bit outside my wheelhouse (which is why I really love LlamaSharp), but I can say that the win-llvm-arm64 build works fine on my Windows ARM laptop.

[martindevans]

The current LLamaSharp version (0.15.0) is compatible with llama.cpp b3479. You need to make sure you're using that version if you're loading custom binaries.

[HillPhelmuth]

The current LLamaSharp version (0.15.0) is compatible with llama.cpp b3479. You need to make sure you're using that version if you're loading custom binaries.

Good to know, thank you.
Ran it again using b3479 version of both win-msvc-arm64 and win-llvm-arm64 and they both load without issue and work just fine with the examples I tried.

[github-actions]

This issue has been automatically marked as stale due to inactivity. If no further activity occurs, it will be closed in 7 days.