Building with -fsycl-targets

[frenchwr]

Hi there, I'm working on creating a deb package of oneMath, which will be built using an already-created deb of DPC++ (version 6.2.0, in a PPA for now) that I am using to build oneMath.

Which of the fsycl-targets from this page are supported with the open source DPC++ compiler?

I ask because for the portFFT backend in oneMath, the build docs suggest using -fsycl-targets option for the best performance on Intel GPUs. Without this flag, oneMath builds fine, but if I try something like -fsycl-targets=spir64_gen or even -fsycl-targets=spir64_x64_64 I get an error:

$ echo "int main(){}" > t.cpp
$ clang++-dpcpp -fsycl -fsycl-targets=spir64_x86_64 t.cpp -o t
llvm-foreach: No such file or directory
clang++-dpcpp: error: x86_64 compiler command failed with exit code 1 (use -v to see invocation)

The build using the default -fsycl-targets=spir64 works fine, so I assume the llvm-foreach error is a red herring, and that instead the problem is that the compiler does not support these targets.

Is that correct? If so, are there special steps I need to follow while building the compiler to enable support for such targets (spir64_x86, spir64_gen)? Or is that a limitation of the open source version of the compiler?

[sarnex]

Hey Will!

All of the listed targets in the link should work, but some require external tools or tools not built by default.

For spir64_x86_64, this is building for an X86 CPU target (not a GPU actually), and requires the opencl-aot tool. I have a decent feeling the llvm-foreach error is really about not being able to find this tool. Basically we want to call opencl-aot on multiple files so we use this llvm-foreach tool to run it on all of them. The error here is not helpful at all, ideally it would print the full command when it fails, I added an enhancement for this.

That tool can be built as part of the build in this repo, but you need to add opencl to a couple of variables in the CMake command.

So a full example build command that builds opencl-aot might look like

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON '-DLLVM_TARGETS_TO_BUILD=host;SPIRV' '-DLLVM_EXTERNAL_PROJECTS=sycl;llvm-spirv;opencl;xpti;xptifw;libdevice;sycl-jit' -DLLVM_EXTERNAL_SYCL_SOURCE_DIR=/tmp/llvm/sycl -DLLVM_EXTERNAL_LLVM_SPIRV_SOURCE_DIR=/tmp/llvm/llvm-spirv -DLLVM_EXTERNAL_XPTI_SOURCE_DIR=/tmp/llvm/xpti -DXPTI_SOURCE_DIR=/tmp/llvm/xpti -DLLVM_EXTERNAL_XPTIFW_SOURCE_DIR=/tmp/llvm/xptifw -DLLVM_EXTERNAL_LIBDEVICE_SOURCE_DIR=/tmp/llvm/libdevice -DLLVM_EXTERNAL_SYCL_JIT_SOURCE_DIR=/tmp/llvm/sycl-jit '-DLLVM_ENABLE_PROJECTS=clang;sycl;llvm-spirv;opencl;xpti;xptifw;libdevice;sycl-jit' -DSYCL_BUILD_PI_HIP_PLATFORM=AMD -DLLVM_BUILD_TOOLS=ON -DLLVM_ENABLE_ZSTD=ON -DLLVM_USE_STATIC_ZSTD=ON -DSYCL_ENABLE_WERROR=OFF -DCMAKE_INSTALL_PREFIX=/tmp/llvm/build/install -DSYCL_INCLUDE_TESTS=ON -DLLVM_ENABLE_DOXYGEN=OFF -DLLVM_ENABLE_SPHINX=OFF -DBUILD_SHARED_LIBS=OFF -DSYCL_ENABLE_XPTI_TRACING=ON -DLLVM_ENABLE_LLD=OFF -DLLVM_SPIRV_ENABLE_LIBSPIRV_DIS=OFF -DXPTI_ENABLE_WERROR=OFF -DSYCL_CLANG_EXTRA_FLAGS= '-DSYCL_ENABLE_BACKENDS=opencl;level_zero;level_zero_v2' -DSYCL_ENABLE_EXTENSION_JIT=ON -DSYCL_ENABLE_MAJOR_RELEASE_PREVIEW_LIB=ON -DBUG_REPORT_URL=https://github.com/intel/llvm/issues /tmp/llvm/llvm

I took that command from our configure.py CMake command generation script so it's not exactly pretty, but hopefully you can see the difference with your current command and take what's needed. The important parts should be

-DLLVM_EXTERNAL_PROJECTS="...;opencl;..."
-DLLVM_ENABLE_PROJECTS="...;opencl;..."
where `...` is any other values you may have for those variables already.

For spir64_gen, that compiles for an Intel GPU ahead of time, so it's used when it's known what the runtime GPU will be. That mode requires the ocloc tool, which is not in this repo and is part of the userspace graphics driver at https://github.com/intel/compute-runtime. A quick look of the CMake file there leads me to believe it is built by default, but I'm not positive.

Let me know if you have any other questions.

Hope this helps.

[frenchwr]

Thanks, @sarnex ! I'll give those steps a try and let you know how it goes.

For spir64_gen, that compiles for an Intel GPU ahead of time, so it's used when it's known what the runtime GPU will be. That mode requires the ocloc tool, which is not in this repo and is part of the userspace graphics driver at https://github.com/intel/compute-runtime. A quick look of the CMake file there leads me to believe it is built by default, but I'm not positive.

We have compute runtime packaged in Ubuntu already, it looks like ocloc comes from the intel-ocloc debian package.

So if I want to build a version of oneMath for Ubuntu that works across Intel CPU and GPU, will -fsycl-targets=spir64_x86_64,spir64_gen give me the desired behavior? Or are you saying that spir64_gen will generate code that is optimized for a GPU it finds at build-time? If so, how can I generate a generic binary that works across all Intel GPUs?

[sarnex]

So if I want to build a version of oneMath for Ubuntu that works across Intel CPU and GPU, will -fsycl-targets=spir64_x86_64,spir64_gen give me the desired behavior? Or are you saying that spir64_gen will generate code that is optimized for a GPU it finds at build-time? If so, how can I generate a generic binary that works across all Intel GPUs?

When using spir64_gen, you need to specify a GPU at compile time, so it's not generic. If you want to build a binary that works on all Intel GPUs, use spir64. For a binary that can run on (all?) CPUs and all Intel GPUs, you would have -fsycl-targets=spir64,spir64_x86_64. The order of values doesn't matter.

[frenchwr]

We were actually already producing opencl-aot in the build, we just didn't identify it as a necessary component. It's now included in our debian package and now with my simple test building with the spir64_x86_64 target I get:

$ clang++-dpcpp -fsycl -fsycl-targets=spir64_x86_64 t.cpp -o t
OpenCL platform ID is empty
OpenCL platform name is empty
Failed to find any of these OpenCL platforms:
  Intel(R) OpenCL
  Intel(R) CPU Runtime for OpenCL(TM) Applications
llvm-foreach: 
clang++-dpcpp: error: x86_64 compiler command failed with exit code 14 (use -v to see invocation)

Is that because only GPU-based devices are detected devices in my environment?

$ sycl-ls 
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Arc(TM) B580 Graphics 20.1.0 [1.6.34666]
[opencl:gpu][opencl:0] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) B580 Graphics OpenCL 3.0 NEO  [25.31.034666]

I think in the past I have seen an OpenCL-based CPU device listed in this output (even with our debian package builds). Is that something that is missing in my environment or do newer versions of the compiler move away from this?

Perhaps I should not be targeting spir64_x86_64 and just rely on the default spir64 (which works fine)?

[sarnex]

Yeah there is a separate OpenCL CPU runtime package that will add an [opencl:cpu] entry in the sycl-ls output.
If you only care about GPUs you can just use spir64 (actually you don't need to pass -fsycl-targets at all).
If you need to support both CPUs and GPUs I can help you figure out what's going on with that error.

[frenchwr]

Ideally we would support CPU and GPU but it's probably not essential. I'm fairly familiar with OpenCL packages in Ubuntu and I'm fairly certain everything installed.

intel-ocloc-dev/resolute,now 25.31.34666.3-1ubuntu1 amd64 [installed]
intel-ocloc-legacy/resolute,now 24.35.30872.40-1 amd64 [installed,automatic]
intel-ocloc/resolute,now 25.31.34666.3-1ubuntu1 amd64 [installed]
ocl-icd-libopencl1/resolute,now 2.3.4-1 amd64 [installed,automatic]
intel-opencl-icd/resolute,now 25.31.34666.3-1ubuntu1 amd64 [installed,automatic]
libopencl-clang14/resolute,now 14.0.2-3 amd64 [installed,automatic]
ocl-icd-opencl-dev/resolute,now 2.3.4-1 amd64 [installed]

Do you see anything that is obviously missing?

I'm getting the following from clinfo:

clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform

[sarnex]

That looks correct for the GPU runtime and OpenCL in general, but I don't see anything that looks like the CPU runtime. In the Intel oneAPI Ubuntu repo it seems the package is called intel-oneapi-runtime-opencl. I don't think this project is open-source, so I don't think it's packaged in the official repos and maybe you don't care about supporting it.

If you just want to try locally, the binaries we use in CI here are available here, but I don't know if this is an official release. I think it depends on TBB also. I have no idea what the license on those binaries is so I wouldn't package it.