3.28.1: why is build own copy of cmake? 🤔

[kloczek]

build output

+ /usr/bin/python3 -sBm build -w --no-isolation
* Getting build dependencies for wheel...
running egg_info
creating src/cmake.egg-info
writing src/cmake.egg-info/PKG-INFO
writing dependency_links to src/cmake.egg-info/dependency_links.txt
writing entry points to src/cmake.egg-info/entry_points.txt
writing requirements to src/cmake.egg-info/requires.txt
writing top-level names to src/cmake.egg-info/top_level.txt
writing manifest file 'src/cmake.egg-info/SOURCES.txt'
ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
WARNING setuptools_scm._file_finders.git git archive detected - fallback to listing all files
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*' found under directory '_skbuild'
warning: no previously-included files matching '*' found under directory 'CMake-src'
adding license file 'LICENSE_Apache_20'
adding license file 'LICENSE_BSD_3'
adding license file 'AUTHORS.rst'
writing manifest file 'src/cmake.egg-info/SOURCES.txt'
* Building wheel...


--------------------------------------------------------------------------------
-- Trying 'Ninja' generator
--------------------------------
---------------------------
----------------------
-----------------
------------
-------
--
Not searching for unused variables given on the command line.
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- The C compiler identification is GNU 14.0.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The CXX compiler identification is GNU 14.0.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_cmake_test_compile/build
--
-------
------------
-----------------
----------------------
---------------------------
--------------------------------
-- Trying 'Ninja' generator - success
--------------------------------------------------------------------------------

Configuring Project
  Working directory:
    /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build
  Command:
    /usr/bin/cmake /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3 -G Ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-install/src/cmake/data -DPYTHON_VERSION_STRING:STRING=3.8.18 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/usr/lib/python3.8/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/usr/bin/python3 -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPYTHON_LIBRARY:PATH=/usr/lib64/libpython3.8.so -DPython_EXECUTABLE:PATH=/usr/bin/python3 -DPython_ROOT_DIR:PATH=/usr -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPython3_EXECUTABLE:PATH=/usr/bin/python3 -DPython3_ROOT_DIR:PATH=/usr -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.8 -DCMAKE_BUILD_TYPE:STRING=Release

Not searching for unused variables given on the command line.
-- The CXX compiler identification is GNU 14.0.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- ***************************************************
-- Build CMake from source: ON
-- ***************************************************
CMake Warning (dev) at /usr/share/cmake/Modules/ExternalProject.cmake:3195 (message):
  The DOWNLOAD_EXTRACT_TIMESTAMP option was not given and policy CMP0135 is
  not set.  The policy's OLD behavior will be used.  When using a URL
  download, the timestamps of extracted files should preferably be that of
  the time of extraction, otherwise code that depends on the extracted
  contents might not be rebuilt if the URL changes.  The OLD behavior
  preserves the timestamps from the archive instead, but this is usually not
  what you want.  Update your project to the NEW behavior or specify the
  DOWNLOAD_EXTRACT_TIMESTAMP option with a value of true to avoid this
  robustness issue.
Call Stack (most recent call first):
  /usr/share/cmake/Modules/ExternalProject.cmake:4418 (_ep_add_download_command)
  CMakeLists.txt:138 (ExternalProject_add)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- SuperBuild - CMakeProject-src-download
-- SuperBuild - CMakeProject-src-download - URL: https://github.com/Kitware/CMake/releases/download/v3.28.3/cmake-3.28.3.tar.gz
-- SuperBuild - CMakeProject-src-download - CMakeProject_SOURCE_DIR: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/CMake-src
-- Looking for __GLIBC__
-- Looking for __GLIBC__ - found
-- Looking for __GLIBC_MINOR__
-- Looking for __GLIBC_MINOR__ - found
-- SuperBuild -   CMakeProject-build
-- SuperBuild -   CMakeProject-build - CMakeProject_BINARY_DIR: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build/CMakeProject-build
-- SuperBuild -   CMakePythonDistributions
-- Configuring done (0.8s)
-- Generating done (0.0s)
-- Build files have been written to: /home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build
[1/27] Performing download step (download, verify and extract) for 'CMakeProject-src-download'
-- Downloading...
   dst='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build/cmake-3.28.3.tar.gz'
   timeout='none'
   inactivity timeout='none'
-- Using src='https://github.com/Kitware/CMake/releases/download/v3.28.3/cmake-3.28.3.tar.gz'
-- [download 0% complete]
-- [download 1% complete]
-- [download 2% complete]
-- [download 3% complete]
-- [download 4% complete]
-- [download 5% complete]
-- [download 6% complete]
-- [download 7% complete]
-- [download 8% complete]
-- [download 9% complete]
-- [download 10% complete]
-- [download 11% complete]
-- [download 12% complete]
-- [download 13% complete]
-- [download 14% complete]
-- [download 15% complete]
-- [download 16% complete]
-- [download 17% complete]
-- [download 18% complete]
-- [download 19% complete]
-- [download 20% complete]
-- [download 21% complete]
-- [download 22% complete]
-- [download 23% complete]
-- [download 24% complete]
-- [download 25% complete]
-- [download 26% complete]
-- [download 27% complete]
-- [download 28% complete]
-- [download 29% complete]
-- [download 30% complete]
-- [download 31% complete]
-- [download 32% complete]
-- [download 33% complete]
-- [download 34% complete]
-- [download 35% complete]
-- [download 36% complete]
-- [download 37% complete]
-- [download 38% complete]
-- [download 39% complete]
-- [download 40% complete]
-- [download 41% complete]
-- [download 42% complete]
-- [download 43% complete]
-- [download 44% complete]
-- [download 45% complete]
-- [download 46% complete]
-- [download 47% complete]
-- [download 48% complete]
-- [download 49% complete]
-- [download 50% complete]
-- [download 51% complete]
-- [download 52% complete]
-- [download 53% complete]
-- [download 54% complete]
-- [download 55% complete]
-- [download 56% complete]
-- [download 57% complete]
-- [download 58% complete]
-- [download 59% complete]
-- [download 60% complete]
-- [download 61% complete]
-- [download 62% complete]
-- [download 63% complete]
-- [download 64% complete]
-- [download 65% complete]
-- [download 66% complete]
-- [download 67% complete]
-- [download 68% complete]
-- [download 69% complete]
-- [download 70% complete]
-- [download 71% complete]
-- [download 72% complete]
-- [download 73% complete]
-- [download 74% complete]
-- [download 75% complete]
-- [download 76% complete]
-- [download 77% complete]
-- [download 78% complete]
-- [download 79% complete]
-- [download 80% complete]
-- [download 81% complete]
-- [download 82% complete]
-- [download 83% complete]
-- [download 84% complete]
-- [download 85% complete]
-- [download 86% complete]
-- [download 87% complete]
-- [download 88% complete]
-- [download 89% complete]
-- [download 90% complete]
-- [download 91% complete]
-- [download 92% complete]
-- [download 93% complete]
-- [download 95% complete]
-- [download 96% complete]
-- [download 98% complete]
-- [download 99% complete]
-- [download 100% complete]
-- verifying file...
       file='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build/cmake-3.28.3.tar.gz'
-- Downloading... done
-- extracting...
     src='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/_skbuild/linux-x86_64-3.8/cmake-build/cmake-3.28.3.tar.gz'
     dst='/home/tkloczko/rpmbuild/BUILD/cmake-python-distributions-3.28.3/CMake-src'
-- extracting... [tar xfz]
-- extracting... [analysis]
-- extracting... [rename]
-- extracting... [clean up]
-- extracting... done

[.and so on.]

Why it is not checked cmake command availability add is added only wrapper on system cmake command which checks only version of the system cmake? 🤔
It does not make ANY sense.

[henryiii]

Wheels can not have any logic in them. They simply are unzipped to the correct directories. And what are the chances the system will have exactly the same version of cmake that you are installing? Or would you expect pip install cmake==3.28.3 to be happy with 3.26.1 or whatever is on your system?

I think what you want is a smart build backend that only adds cmake if the command is either not present or not of sufficient version - which is exactly what scikit-build-core does!

[kloczek]

Wheels can not have any logic in them. They simply are unzipped to the correct directories. And what are the chances the system will have exactly the same version of cmake that you are installing? Or would you expect pip install cmake==3.28.3 to be happy with 3.26.1 or whatever is on your system?

I'm expecting that this module would be just wrapper checking cmake executable version according to its version and that's it ..

I think what you want is a smart build backend that only adds cmake if the command is either not present or not of sufficient version - which is exactly what scikit-build-core does!

I don't see at all building own copy of cmake it it provides only interface to use cmake.
The same pkgconfig python module (https://github.com/matze/pkgconfig/) is not building own copy of the pkgconfig.
It provides only python interface to use it.

[henryiii]

The pkgconfig module is not a PyPI copy of pkg-config, it's a wrapper around a command, just like scikit-build-core is a wrapper around the cmake command providing a custom Python interface. If you don't have pkg-config pre-installed, then pkgconfig module raises an error. The version of pkgconfig (Python) has nothing to do with the version of peg-config you've pre-installed.

The entire point of the cmake module is to distribute cmake. It does not wrap the command (other than providing a very, very simple and not quite perfect cmake runner wrapper to make it easier to get the paths right).

cmake executable version according to its version

The chances of it exactly matching are really small. If you have 3.28.1 and the cmake module is 3.28.2, then it would still have to build, since you requested 3.28.2. We don't release every patch release, so some system CMake's wouldn't be possible to request a "dummy" for at all. And what do you do if cmake is present but ctest is missing?

What's worse, the builder can't tell if you are installing or making a wheel (modern packaging - there used to be a setup.py install but it's replaced by always going through a wheel). If you are preparing the wheels for offline use (such as in test suites - scikit-build-core does exactly this in it's tests, HPC/grid/cloud computing, etc), then how do you know that you can't just make the "dummy" cmake package if you happen to have the matching CMake version present? Would your CI just randomly break if you upgrade CMake? Also, if you get a universal wheel, you'd expect a universal CMake even if you only have x86 or ARM installed.

Python packaging is not compatible with this sort of design. A package should not build different wheels based on its environment. A user expects to get the version of CMake they asked for. And you can't tell what a user actually asked for! cmake > 3.15 will resolve to cmake-3.28.3.tar.gz, and there's no way to then backtrack and tell that the system CMake (say, 3.17) is actually fine according to what the user asked. And once you've installed it, there's no way to re-evalutate it if someone then askes for cmake > 3.21, the system things that 3.28.3 is installed even if it's a dummy package for 3.17!

There's a very simple and easy solution for building package with CMake: Only ask for the Python package if you need a copy of cmake! As I've said, scikit-build-core does this, and there's an example in scikit-build's docs on how to do it yourself, it's less LoC than most setuptools customizations used to be. meson-python does this with ninja and patchelf too, which are also binary redistributions on PyPI.

This was discussed for ninja-python-distributions multiple times with the same conclusion: fix the builders to only ask for what they need instead of making lying packages that don't include what they are supposed to include based on the system they were built on.

By the way, 95% of the time you should get a binary; nothing will build. We provide wheels for every system we can. It only builds if you make it build (as shown above) or if you are on a system like Pyodide / FreeBSD / Android / iOS, where we can't provide wheels. Again, the fix is easy - only request it in get_requires_for_build_wheel() if cmake is not present or not in the version range you need. scikit-build-core 0.8+ even lets you specify a version range or disallow specific versions (any valid Python version specifiers allowed).

[kloczek]

By the way, 95% of the time you should get a binary; nothing will build. We provide wheels for every system we can. It only builds if you make it build (as shown above) or if you are on a system like Pyodide / FreeBSD / Android / iOS, where we can't provide wheels. Again, the fix is easy - only request it in get_requires_for_build_wheel() if cmake is not present or not in the version range you need. scikit-build-core 0.8+ even lets you specify a version range or disallow specific versions (any valid Python version specifiers allowed).

Waste of time.
cmake is now peaty stable and is available as REGULAR and NATIVE package on all those systems.
New cmake releases are only providing some minor cleanups/bugfixes and release this module ne version only because new cmake version has been released is nothing more than another waste of time.
Look on last this module releases diffs and you will find that actually NOTHING has changed in actual cmake interface.
All changes are related to downloading new versions of source code.

[henryiii]

I catalog changes in cmake on this page: https://cliutils.gitlab.io/modern-cmake/chapters/intro/newcmake.html

And the versions available on various systems on this page: https://cliutils.gitlab.io/modern-cmake/chapters/intro/installing.html

And there are massive, huge changes in each version. 3.28 added support for C++20 modules. So if I want to use those, I have to set cmake>=3.28. The first version to properly support FindPython was 3.18.2 (with important changes as recently as 3.26, like support for Stable ABI), which you'll find is not available on even the newest LTS Ubuntu. And the next version coming out (RC1 came out a couple of days ago) will finally have the ability to make the test target depend on ALL!

If you want to make a "cmake-if-needed" SDist only package, go ahead. There's nothing stopping you. But we can't break everyone (including most scikit-build-core users!) who need a modern CMake and request it (scikit-build-core requests it only if actually needed, responsibly!) by not shipping cmake in a cmake package. No one is asking ruff, clang-format, or any of the other built CLI tools to ignore distributing ruff or clang-format binaries if there is already a matching copy on the system (for the long list of reasons above, it would be a terrible idea).

This is not the right solution to the problem. The correct solution is to fix those packages to only request cmake if the system doesn't provide one. The mechanism for doing exactly that was standardized in 2016.

As I said on the ninja discussions, I'd also be okay with an opt-in way to make a dummy package, but we are about to move the build backend, so it would be best to wait on that. I also want to change the way the binary is provided, which will happen after the move but before any dummy option.

[henryiii]

Also, 3.27 removed (sort-of) the old FindPythonLibs/FindPythonInterp. So the CMake version really matters to Python users.

[kloczek]

And there are massive, huge changes in each version. 3.28 added support for C++20 modules. So if I want to use those, I have to set cmake>=3.28. The first version to properly support FindPython was 3.18.2 (with important changes as recently as 3.26, like support for Stable ABI), which you'll find is not available on even the newest LTS Ubuntu. And the next version coming out (RC1 came out a couple of days ago) will finally have the ability to make the test target depend on ALL!

Please show me ONE python module which depends on some cmake features added after first cmake 3.0.
I have already packaged as rpm packages

[tkloczko@pers-jacek SPECS]$ ls python-*spec -1| wc -l
1245

Amongst those modules there ONLY SIX

[tkloczko@pers-jacek SPECS]$ grep "BuildRequires:.*cmake$" python-*
python-gcovr.spec:BuildRequires:        cmake
python-pybind11.spec:BuildRequires:     cmake
python-rapidfuzz.spec:BuildRequires:    cmake
python-scikit-build-core.spec:BuildRequires:    cmake
python-scikit-build.spec:BuildRequires: cmake
python-ssh-python.spec:BuildRequires:   cmake

which are using cmake and none are using your module.
NONE of those modules depends on any cmake features added after first cmake 3.x release.

cmake 3.18.2 has been released almost FOUR years ago. It was the time when still python 2.x was in use,
And look on stats. ~1.245k python modules and less than HALF percent uses now cmake.

[burgholzer]

Please show me ONE python module which depends on some cmake features added after first cmake 3.0.
I have already packaged as rpm packages

A little more than one example: All our C++ projects with Python bindings (e.g., https://github.com/cda-tum/mqt-core) require CMake>=3.19 and might, at some point, jump to 3.24 for better FetchContent support.

[kloczek]

Why do you want to use FetchContent if cmake already is installed in the system image? 🤔

[burgholzer]

Why do you want to use FetchContent if cmake already is installed in the system image? 🤔

We do not use FetchContent to get CMake, but other project dependencies.
CMake is just provided via the mechanisms in scikit-build-core described above by Henry.

[kloczek]

We do not use FetchContent to get CMake, but other project dependencies.

Name of those modules.
None of those six modules which I've mentioned so far which are using cmake are using FetchContent.
Other fact: do you know that MAJORITY of the most frequently used Linux distributions production build envs are INTENTIONALLLY cut off from access to the public network?
How do you recommend to perform build of your module in such conditions? 🤔

[burgholzer]

Name of those modules. None of those six modules which I've mentioned so far which are using cmake are using FetchContent.

For example, googletest or nlohmann_josn to name the most prominent ones.

Other fact: do you know that MAJORITY of the most frequently used Linux distributions production build envs are INTENTIONALLLY cut off from access to the public network? How do you recommend to perform build of your module in such conditions? 🤔

By making use of the modern find_package integration of FetchContent that landed in 3.24 (which is manually back ported to 3.19 in our projects). See https://cmake.org/cmake/help/latest/module/FetchContent.html#integrating-with-find-package

[kloczek]

Name of those modules. None of those six modules which I've mentioned so far which are using cmake are using FetchContent.

For example, googletest or nlohmann_josn to name the most prominent ones.

https://pypi.org/project/gtest/ is no longer maintained. Homepage shows 404.

https://github.com/nlohmann/json/ it is not python module.

Other fact: do you know that MAJORITY of the most frequently used Linux distributions production build envs are INTENTIONALLLY cut off from access to the public network? How do you recommend to perform build of your module in such conditions? 🤔

By making use of the modern find_package integration of FetchContent that landed in 3.24 (which is manually back ported to 3.19 in our projects). See https://cmake.org/cmake/help/latest/module/FetchContent.html#integrating-with-find-package

I've not been asking for what is FetchContent and with what it is integrated.
Sorry but please read again my question.

[burgholzer]

https://pypi.org/project/gtest/ is no longer maintained. Homepage shows 404.

https://github.com/nlohmann/json/ it is not python module.

Yeah. And I never claimed that. These are dependencies of our C++ code that is then exposed as a Python module via pybind11 and distributed to PyPI.
So, although these are dependencies of our C++ code, they are also dependencies of the resulting Python module.
And building that Python module requires CMake>=3.19.

I've not been asking for what is FetchContent and with what it is integrated. Sorry but please read again my question.

... 🙃

Other fact: do you know that MAJORITY of the most frequently used Linux distributions production build envs are INTENTIONALLLY cut off from access to the public network?

Yes.

How do you recommend to perform build of your module in such conditions? 🤔

By having the required dependencies installed on the system before the build. FetchContent will directly pick up the installed version without ever accessing the public network.

[kloczek]

How do you recommend to perform build of your module in such conditions? 🤔

By having the required dependencies installed on the system before the build. FetchContent will directly pick up the installed version without ever accessing the public network.

So WHY you don't require have installed cmake before build your module? 🤔