Minimal working example linking the wheel statically

[peekxc]

Is it possible to create a minimal, cross-platform example using meson that properly links the wheel?

I have a need to call BLAS/LAPACK routines from within a C++ file included as part of a Python package. While I can get varying such implementations linked with my own machine, I'm finding it difficult to do so in a portable manner, which lead me here.

Ideally, I would like to be able to just specify scipy-openblas32 as a build dependency in some given package's pyproject.toml and then link to it as necessary within a meson.build file using meson-python.

Using some of numpy's tools (+this advice), I created test repository that generates a pkg-config file suitable for a meson-dependency object to use, but when I use these wheels unfortunately I get various linker errors. I can fix the build myself for my own machine, but this eliminates the portability.

Is there any "best practices" advice surrounding how to do this?

[mattip]

I get various linker errors

Could you add a CI run that shows the problem or attach a log? What platform, what version of meson?

[peekxc]

I'll try to modify the package to be a bonafide Python package that I can test CI runs with.

What platform, what version of meson?

While I'm working with meson 1.2.1 on OSX 10.15 (x86_64), I guess I was more wondering what the general technique is to link BLAS across e.g. all the builds targeted by cibuildwheel.

I've looked at SciPy's meson.build files, in particular these lines, SciPy manages to link these wheels via:

# First try scipy-openblas, and if found don't look for cblas or lapack, we
# know what's inside the scipy-openblas wheels already.
if blas_name == 'openblas' or blas_name == 'auto'
  blas = dependency('scipy-openblas', required: false)
  if blas.found()
    blas_name = 'scipy-openblas'
  endif
endif
...
if blas_name == 'scipy-openblas'
  python_sources += ['_distributor_init_local.py']
endif

That seems to be the only references in scipy-openblas I could find. I tried replicating most of this, including the automatic generation of a pc file, but to no avail. In particular, when I do:

PKG_CONFIG_PATH=.openblas meson setup --wipe build

Everything seems normal:

The Meson build system
Version: 1.2.1
Source dir: /Users/mpiekenbrock/lapack_mre
Build dir: /Users/mpiekenbrock/lapack_mre/build
Build type: native build
Project name: hello
Project version: undefined
C++ compiler for the host machine: /usr/local/opt/llvm/bin/clang++ (clang 15.0.7 "Homebrew clang version 15.0.7")
C++ linker for the host machine: /usr/local/opt/llvm/bin/clang++ ld64 609.8
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 found: YES (/Users/mpiekenbrock/opt/miniconda3/envs/spri/bin/python)
Found pkg-config: /usr/local/bin/pkg-config (0.29.2)
Run-time dependency python found: YES 3.11
Message: Python path =/Users/mpiekenbrock/opt/miniconda3/envs/spri/bin/python
Message: Numpy version =1.26.1
Fortran compiler for the host machine: gfortran (gcc 12.2.0 "GNU Fortran (Homebrew GCC 12.2.0) 12.2.0")
Fortran linker for the host machine: gfortran ld64 609.8
Compiler for Fortran supports arguments -Wno-conversion: YES 
Run-time dependency scipy-openblas found: YES 0.3.23.dev
Build targets in project: 1

Found ninja-1.11.1.git.kitware.jobserver-1 at /Users/mpiekenbrock/opt/miniconda3/envs/spri/bin/ninja

But then when I cd into the build directory and run meson compile, I get the classic undefined symbols:

$ meson compile 
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /Users/mpiekenbrock/opt/miniconda3/envs/spri/bin/ninja
[2/2] Linking target hello_lapack
FAILED: hello_lapack 
/usr/local/opt/llvm/bin/clang++  -o hello_lapack hello_lapack.p/hello.cpp.o -L/Users/mpiekenbrock/opt/miniconda3/envs/spri/lib/python3.11/site-packages/scipy_openblas64/lib -L/usr/local/lib -L/usr/local/opt/llvm/lib -I/usr/local/opt/llvm/include -Wl,-dead_strip_dylibs -Wl,-headerpad_max_install_names -Wl,-undefined,error -lm -lpthread -lgfortran -lquadmath -lopenblas_python
Undefined symbols for architecture x86_64:
  "_dgemm_", referenced from:
      _main in hello.cpp.o
  "_dsyev_", referenced from:
      _main in hello.cpp.o
ld: symbol(s) not found for architecture x86_64
clang-15: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.

For reference, the file I'm trying to compile is just a toy .cpp file that tries to call dgemm and dsyev on a simple matrix. The only relevant lines are:

#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <cmath>

//definition for lapack / blas routines
#ifdef __cplusplus
extern "C" {
#endif
  int dgemm_( char *transa, char *transb, const int *m, const int *n, const int *k,
            double *alpha, const double *a, const int *lda, const double *b,
            const int *ldb, double *beta, double *c, const int *ldc);
  int dsyev_( char *jobz, char *uplo, int *n, double *a, int *lda, double *w,
            double *work, int *lwork, int *info);
#ifdef __cplusplus
}
#endif

[mattip]

I see the link argument -lopenblas_python, and the link succeeds, so it seems it should JustWork. Why not use #include <cblas.h> instead of rolling your own declarations? That way you might see a clearer error when the declarations do not line up with what is in the binary library/shared object.

[mattip]

FWIW, going forward you will need to add a scipy_ prefix to the symbols.

[peekxc]

I'll keep that in mind / try it sometime. I ended giving up after spending too many hours on it, it seems linking to BLAS / LAPACK functions in C++ in a simple, cross-platform, CI-compatible way such that I can specify SciPy/NumPy as a dependency in a Python package and then make native LAPACK calls in C++ is just too difficult a task to achieve!

I'll keep perusing how SciPy / NumPy handle things internally, but am otherwise just going to wait and watch e.g. 20002 and 17244

[ev-br]

Speaking of scipy/scipy#20002, you can do scipy/scipy#20002 (comment) .

The magic sauce seems to be including scipy/_build_utils/src/npy_cblas.h (which is IIUC, a nearly verbatim copy of a numpy header file) and using the BLAS_FUNC macro from it, which does all the name mangling. You'll still need to add the prototype of course, using the mangled name: scipy/scipy@39f8344#diff-1d06e2703e983014d899d2c42544507f537a5c6eebc84ec484cf125eed03e102R79

The PR this is from is https://github.com/scipy/scipy/pull/19970/commits