[libc] Fix libc GPU bitcode entrypoint library build on Windows

[jinge90]

This PR aims to solve: #209381
On Windows, if using 'add_executable' to build final LLVM IR module for GPU targets, cmake will regard current target as normal Windows .exe or .dll file adding a bunch of flags which GPU toolchain can't handle, this will lead to final linking step abort. For example, we encounter following error message for amdgcn target:
llvm-project\build.\bin\clang++.exe --target=amdgcn-amd-amdhsa -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wno-comment -Wstring-conversion -Wno-pass-failed -ffunction-sections -fdata-sections -Xclang -fno-pch-timestamp -O3 -DNDEBUG -Xlinker --stack -Xlinker 16777216 --target=amdgcn-amd-amdhsa -r -nostdlib -flto -Wl,--lto-emit-llvm @CMakeFiles\libmbitcode.rsp -o libc\lib\libm.bc.exe -Wl,--out-implib,libc\lib\liblibm.bc.dll.a -Wl,--major-image-version,0,--minor-image-version,0 && cd ."
ld.lld: error: unknown argument '--stack'
ld.lld: error: unknown argument '--major-image-version'
ld.lld: error: unknown argument '--minor-image-version'
ld.lld: error: cannot open 16777216: no such file or directory
ld.lld: error: unable to find library -lkernel32
ld.lld: error: unable to find library -luser32
ld.lld: error: unable to find library -lgdi32
ld.lld: error: unable to find library -lwinspool
ld.lld: error: unable to find library -lshell32
ld.lld: error: unable to find library -lole32
ld.lld: error: unable to find library -loleaut32
ld.lld: error: unable to find library -luuid
ld.lld: error: unable to find library -lcomdlg32
ld.lld: error: unable to find library -ladvapi32

We follow libclc's approach to use llvm-link to build final LLVM IR module for Windows host, the idea is building static archive library firstly and then use it as input to llvm-link.

[llvmorg-github-actions]

@llvm/pr-subscribers-github-workflow

@llvm/pr-subscribers-libc

Author: jinge90

Changes

---

Full diff: https://github.com/llvm/llvm-project/pull/210662.diff

4 Files Affected:

• (modified) libc/CMakeLists.txt (+7)
• (modified) libc/cmake/modules/LLVMLibCLibraryRules.cmake (+55-5)
• (modified) libc/lib/CMakeLists.txt (+13-5)
• (modified) libc/startup/gpu/CMakeLists.txt (+16-8)

diff --git a/libc/CMakeLists.txt b/libc/CMakeLists.txt
index af820ba6ab6c0..3643ef7213a81 100644
--- a/libc/CMakeLists.txt
+++ b/libc/CMakeLists.txt
@@ -289,6 +289,13 @@ endif()
 if(LIBC_TARGET_OS_IS_GPU)
   include(prepare_libc_gpu_build)
   set(LIBC_ENABLE_UNITTESTS OFF)
+
+  if(CMAKE_HOST_WIN32)
+    find_program(LIBC_LLVM_LINK llvm-link PATHS ${LLVM_TOOLS_BINARY_DIR} NO_DEFAULT_PATH)
+    if(NOT LIBC_LLVM_LINK)
+      message(FATAL_ERROR "llvm-link not found in ${LLVM_TOOLS_BINARY_DIR}")
+    endif()
+  endif()
 elseif(LIBC_TARGET_OS_IS_BAREMETAL)
   set(LIBC_ENABLE_UNITTESTS OFF)
 endif()
diff --git a/libc/cmake/modules/LLVMLibCLibraryRules.cmake b/libc/cmake/modules/LLVMLibCLibraryRules.cmake
index 2df8f49100b99..4e6532a4402cb 100644
--- a/libc/cmake/modules/LLVMLibCLibraryRules.cmake
+++ b/libc/cmake/modules/LLVMLibCLibraryRules.cmake
@@ -80,13 +80,43 @@ function(get_all_object_file_deps result fq_deps_list)
   set(${result} ${all_deps} PARENT_SCOPE)
 endfunction()
 
-# A rule to build a library from a collection of entrypoint objects and bundle
-# it in a single LLVM-IR bitcode file.
+# Link bitcode inputs into a single LLVM IR module using llvm-link. Used on
+# Windows where 'add_executable' can't work. Build static archive library
+# firstly and use this static archive library as llvm-link's input.
 # Usage:
-#     add_bitcode_entrypoint_library(
+#     llvm_link_bitcode(
+#       target_name
+#       OUTPUT <output file path>
+#       INPUTS <input file path>
+#       DEPENDS <targets that must be built first>
+#     )
+function(llvm_link_bitcode target_name)
+  cmake_parse_arguments(
+    "ARG"
+    ""
+    "OUTPUT"
+    "INPUTS;DEPENDS"
+    ${ARGN}
+  )
+  add_custom_command(
+    OUTPUT ${ARG_OUTPUT}
+    COMMAND ${LIBC_LLVM_LINK} ${ARG_INPUTS} -o ${ARG_OUTPUT}
+    DEPENDS ${ARG_DEPENDS}
+    COMMENT "Linking bitcode ${ARG_OUTPUT}"
+  )
+  add_custom_target(${target_name} ALL DEPENDS ${ARG_OUTPUT})
+  set_target_properties(${target_name} PROPERTIES TARGET_FILE ${ARG_OUTPUT})
+endfunction(llvm_link_bitcode)
+
+# Build a single LLVM IR module by using clang driver to link object files
+# files with LTO. Used on Linux platform.
+# Usage:
+#     add_bitcode_entrypoint_library_lto(
+#       target_name
+#       base_target_name
 #       DEPENDS <list of add_entrypoint_object targets>
 #     )
-function(add_bitcode_entrypoint_library target_name base_target_name)
+function(add_bitcode_entrypoint_library_lto target_name base_target_name)
   cmake_parse_arguments(
     "ENTRYPOINT_LIBRARY"
     "" # No optional arguments
@@ -112,11 +142,31 @@ function(add_bitcode_entrypoint_library target_name base_target_name)
   if(LIBC_TARGET_ARCHITECTURE_IS_SPIRV)
       target_link_options(${target_name} PRIVATE "${LIBC_COMPILE_OPTIONS_DEFAULT}"
                       "-nostdlib" "-emit-llvm")
-  else()  
+  else()
       target_link_options(${target_name} PRIVATE "${LIBC_COMPILE_OPTIONS_DEFAULT}"
                       "-r" "-nostdlib" "-flto" "-Wl,--lto-emit-llvm")
   endif()
   add_dependencies(${base_target_name} ${target_name})
+endfunction(add_bitcode_entrypoint_library_lto)
+
+# A rule to build a library from a collection of entrypoint objects and bundle
+# it in a single LLVM IR module.
+# Usage:
+#     add_bitcode_entrypoint_library(
+#       target_name
+#       base_target_name
+#       DEPENDS <list of add_entrypoint_object targets>
+#     )
+function(add_bitcode_entrypoint_library target_name base_target_name)
+  if(CMAKE_HOST_WIN32)
+    llvm_link_bitcode(${target_name}
+      OUTPUT ${LIBC_LIBRARY_DIR}/${target_name}.bc
+      INPUTS $<TARGET_FILE:${base_target_name}>
+      DEPENDS ${base_target_name}
+    )
+  else()
+    add_bitcode_entrypoint_library_lto(${target_name} ${base_target_name} ${ARGN})
+  endif()
 endfunction(add_bitcode_entrypoint_library)
 
 # A rule to build a library from a collection of entrypoint objects.
diff --git a/libc/lib/CMakeLists.txt b/libc/lib/CMakeLists.txt
index 8ef0329471521..0029dcabd3ac8 100644
--- a/libc/lib/CMakeLists.txt
+++ b/libc/lib/CMakeLists.txt
@@ -63,11 +63,19 @@ install(
 )
 
 foreach(file ${added_bitcode_targets})
-  install(FILES $<TARGET_FILE:${file}>
-          DESTINATION ${LIBC_INSTALL_LIBRARY_DIR}
-          RENAME $<TARGET_PROPERTY:${file},OUTPUT_NAME>
-          COMPONENT libc
-  )
+  if(CMAKE_HOST_WIN32)
+    install(FILES $<TARGET_PROPERTY:${file},TARGET_FILE>
+            DESTINATION ${LIBC_INSTALL_LIBRARY_DIR}
+            RENAME $<TARGET_PROPERTY:${file},OUTPUT_NAME>
+            COMPONENT libc
+    )
+  else()
+    install(FILES $<TARGET_FILE:${file}>
+            DESTINATION ${LIBC_INSTALL_LIBRARY_DIR}
+            RENAME $<TARGET_PROPERTY:${file},OUTPUT_NAME>
+            COMPONENT libc
+    )
+  endif()
 endforeach()
 
 # Add empty archive libraries for compatibility with systems expecting certain
diff --git a/libc/startup/gpu/CMakeLists.txt b/libc/startup/gpu/CMakeLists.txt
index 12cb891a687d8..cf7ea76a2ba8e 100644
--- a/libc/startup/gpu/CMakeLists.txt
+++ b/libc/startup/gpu/CMakeLists.txt
@@ -27,15 +27,23 @@ function(add_startup_object name)
       OUTPUT_NAME ${name}.o
   )
 
-  # Make an executable target of relocatable bitcode for clang if needed.
+  # Make a relocatable bitcode object for clang if needed.
   if(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR)
-    add_executable(${fq_target_name}.exe $<TARGET_OBJECTS:${fq_target_name}>)
-    set_target_properties(${fq_target_name}.exe PROPERTIES
-      RUNTIME_OUTPUT_DIRECTORY ${LIBC_LIBRARY_DIR}
-      RUNTIME_OUTPUT_NAME ${name}.o)
-    target_link_options(${fq_target_name}.exe PRIVATE
-                        ${LIBC_COMPILE_OPTIONS_DEFAULT}
-                        "-r" "-nostdlib" "-flto" "-Wl,--lto-emit-llvm")
+    if(CMAKE_HOST_WIN32)
+      llvm_link_bitcode(${fq_target_name}.exe
+        OUTPUT ${LIBC_LIBRARY_DIR}/${name}.o
+        INPUTS $<TARGET_OBJECTS:${fq_target_name}>
+        DEPENDS ${fq_target_name}
+      )
+    else()
+      add_executable(${fq_target_name}.exe $<TARGET_OBJECTS:${fq_target_name}>)
+      set_target_properties(${fq_target_name}.exe PROPERTIES
+        RUNTIME_OUTPUT_DIRECTORY ${LIBC_LIBRARY_DIR}
+        RUNTIME_OUTPUT_NAME ${name}.o)
+      target_link_options(${fq_target_name}.exe PRIVATE
+                          ${LIBC_COMPILE_OPTIONS_DEFAULT}
+                          "-r" "-nostdlib" "-flto" "-Wl,--lto-emit-llvm")
+    endif()
   endif()
 endfunction()
 

[jinge90]

Hi, @jhuber6
Could you help review this patch?
This patch uses static archive library as llvm-link's input which aligns with libclc and I have another version of patch which reuses the internal object list from static archive library target, I can switch to that approach if you prefer.
With this patch applied, the errors for Windows will be gone and libc.bc, libm.bc and crt1.o will be generated as expected.
Thanks very much.

[jhuber6]

The easiest change is to make this consume the static library that the previous steps produce. we should make both paths do the same thing

[jinge90]

The easiest change is to make this consume the static library that the previous steps produce. we should make both paths do the same thing

Hi, @jhuber6
Current patch just consumes static archive library as llvm-link input for bitcode generation, may I know what do you mean by "make both paths do the same thing"? Do you mean use this llvm-link approach for both Linux and Windows?
Thanks very much.

[jhuber6]

Hi, @jhuber6 Current patch just consumes static archive library as llvm-link input for bitcode generation, may I know what do you mean by "make both paths do the same thing"? Do you mean use this llvm-link approach for both Linux and Windows? Thanks very much.

No WIN32 split

[jinge90]

Hi, @jhuber6
I found some other issues when testing this patch:

1. we introduce dependency on llvm-link for gpu target, so we need to add it to extra_deps in llvm/runtimes/CMakeLists.txt when LLVM_LIBC_GPU_BUILD

2. After fixing 1, the normal bootstrapping build work as expected but build fails when using cmake cache for gpu, the reason is amd team recently changes the libc target for amd gpu to 'amdgpu-amd-amdhsa' and the LLVM_LIBC_GPU_BUILD flag is set according to this new target name while 'amdgcn-amd-amdhsa' still used in cmake cache for gpu, this lead to LLVM_LIBC_GPU_BUILD not set, so all required extra_deps will be missed.

3. The pre-ci for amd gpu is standalone build and uses a pre-build dock image, llvm-link is not included in this image.
   I wonder whether we should update the dock image for libc testing?
   It looks like we need to firstly update libc Dockfile in https://github.com/llvm/llvm-project/blob/main/.github/workflows/containers/libc/Dockerfile , to install llvm-23 in the docker image to include llvm-link tool.
   After the Dockfile merged and the docker image updated, we can proceed with current PR?
   The PR to update the libc container to install llvm-23 to include llvm-link is: Create symlink for llvm-link in libc docker image #210929

Thanks very much.

[jhuber6]

CMake Error at /__w/llvm-project/llvm-project/libc/CMakeLists.txt:294 (message):
  llvm-link not found in /__w/llvm-project/llvm-project/build/bin or system
  path

Weird, this should be listed as a runtime dependency already I'd think.