No module named 'numpy.core._multiarray_umath' ~ Bazel Edition

[KennyHoang-CS]

Steps to reproduce:

Python 3.10
Numpy 2.0.2

Using aspect bazel rules and bazel module,

import numpy

bazel run /your/oci_load_target
docker run -p 8501:8501 --rm /your repo tag defined in oci load bazel rule/

Error message:

(venv) bazel run //myapp:image_load           
INFO: Analyzed target //myapp:image_load (7 packages loaded, 9499 targets configured).
INFO: Found 1 target...
Target //myapp:image_load up-to-date:
  bazel-bin/myapp/image_load.sh
INFO: Elapsed time: 15.954s, Critical Path: 9.53s
INFO: 7 processes: 3 internal, 4 darwin-sandbox.
INFO: Build completed successfully, 7 total actions
INFO: Running command line: bazel-bin/myapp/image_load.sh
79d13e645cd6: Loading layer [==================================================>]  96.62MB/96.62MB
The image localhost/myapp:latest already exists, renaming the old one with ID sha256:031ee52b0c21e88f76e171e88ce84031a3ea19a9c183b9437979161e38a5e813 to empty string
Loaded image: localhost/myapp:latest
(venv) khoang@CWY7PYM2VD  docker run -p 8501:8501 --rm localhost/myapp:latest
Traceback (most recent call last):
  File "/myapp/myapp.runfiles/.myapp.venv/lib/python3.10/site-packages/numpy/_core/__init__.py", line 23, in <module>
    from . import multiarray
  File "/myapp/myapp.runfiles/.myapp.venv/lib/python3.10/site-packages/numpy/_core/multiarray.py", line 10, in <module>
    from . import overrides
  File "/myapp/myapp.runfiles/.myapp.venv/lib/python3.10/site-packages/numpy/_core/overrides.py", line 8, in <module>
    from numpy._core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy._core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/myapp/myapp.runfiles/.myapp.venv/lib/python3.10/site-packages/numpy/__init__.py", line 114, in <module>
    from numpy.__config__ import show as show_config
  File /myapp/myapp.runfiles/.myapp.venv/lib/python3.10/site-packages/numpy/__config__.py", line 4, in <module>
    from numpy._core._multiarray_umath import (
  File "/myapp/myapp.runfiles/.myapp.venv/lib/python3.10/site-packages/numpy/_core/__init__.py", line 49, in <module>
    raise ImportError(msg)
ImportError: 

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for
many reasons, often due to issues with your setup or how NumPy was
installed.

We have compiled some common reasons and troubleshooting tips at:

    https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

  * The Python version is: Python3.10 from "/myapp/./myapp.runfiles/.myapp.venv/bin/python3"
  * The NumPy version is: "2.0.2"

and make sure that they are the versions you expect.
Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy._core._multiarray_umath'


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/myapp/./myapp.runfiles/_main/myapp/numpyfailing.py", line 1, in <module>
    import numpy as np
  File "/myapp/myapp.runfiles/.myapp.venv/lib/python3.10/site-packages/numpy/__init__.py", line 119, in <module>
    raise ImportError(msg) from e
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python interpreter from there.

Additional information:

BUILD.bazel

load("@rules_oci//oci:defs.bzl", "oci_image", "oci_push", "oci_load")
load("@rules_uv//uv:pip.bzl", "pip_compile")
load("@aspect_rules_py//py:defs.bzl", "py_binary", "py_library")
load("@aspect_bazel_lib//lib:tar.bzl", "tar")
load("@pypi//:requirements.bzl", "requirement")
load("@aspect_bazel_lib//lib:transitions.bzl", "platform_transition_filegroup", "platform_transition_binary")

pip_compile(name = "generate_requirements_txt")

exports_files([
    "requirements.in",
    "requirements.txt",
    "requirements_lock.txt",
    "src/__init__.py",
])

platform(
    name = "aarch64_linux",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:aarch64",
    ],
)

platform(
    name = "x86_64_linux",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
    ],
)

# bazel build //myapp:platform_myapp_binary
platform_transition_binary(
    name = "platform_myapp_binary",
    binary = ":myapp",
    target_platform = select({
        "@platforms//cpu:arm64": ":aarch64_linux",
        "@platforms//cpu:x86_64": ":x86_64_linux",
    }),
)

platform_transition_filegroup(
    name = "platform_image",
    srcs = [":myapp_image"],
    target_platform = select({
        "@platforms//cpu:arm64": ":aarch64_linux",
        "@platforms//cpu:x86_64": ":x86_64_linux",
    }),
)

## see https://bazel.build/reference/be/python#py_library
# bazel build //myapp:myapp_lib
py_library(
    name = "myapp_lib",
    srcs = [
        "numpyfailing.py"
    ],
    deps = [
        "@pypi//numpy",
    ],
)

# bazel build //myapp:myapp
# bazel run //myapp:myapp
py_binary(
    name = "myapp",
    srcs = ["numpyfailing.py"],
    imports = ["."],
    main = "numpyfailing.py",
    visibility = ["//visibility:public"],
    deps = [
        ":myapp_lib",
    ],
)

# bazel build //myapp:myapp_tar
tar(
    name = "myapp_tar",
    srcs = [
        ":myapp",
    ],
    visibility = ["//visibility:public"],
)


# bazel build //myapp:myapp_image
oci_image(
    name = "myapp_image",
    base = "@python_base_oci",
    entrypoint = [
        "/bin/bash",
        "-c",
        "cd /myapp && ./myapp",
    ],
    env = {
        "DD_GIT_REPOSITORY_URL": "{DD_GIT_REPOSITORY_URL}",
        "DD_GIT_COMMIT_SHA": "{DD_GIT_COMMIT_SHA}",
    },
    tars = [":myapp_tar"],
)

# To build the image and load it into it into a local runtime:
# $ bazel run //myapp:image_load
# $ docker run -p 8501:8501 --rm localhost/myapp:latest
oci_load(
    name = "image_load",
    image = ":platform_image",
    repo_tags = ["localhost/myapp:latest"],
)

numpyfailing.py

import numpy as np
import sys

print("Python version:", sys.version)
print("NumPy version:", np.__version__)

# Create a 2x3 array filled with random numbers
array = np.random.rand(2, 3)
print("Random 2x3 Array:")
print(array)

# Perform basic operations
print("\nArray Transpose:")
print(array.T)

print("\nSum of all elements:")
print(np.sum(array))

print("\nMean of all elements:")
print(np.mean(array))

print("\nElement-wise multiplication:")
print(array * 2)

MODULE.bazel

module(
    name = "numpy fail demo",
    version = "1.0.0",
)

"Bazel module dependencies, see https://bazel.build/external/overview#bzlmod"

bazel_dep(name = "aspect_rules_lint", version = "1.2.2")
bazel_dep(name = "aspect_bazel_lib", version = "2.14.0")
bazel_dep(name = "aspect_rules_py", version = "1.3.2", dev_dependency = True)
bazel_dep(name = "bazel_skylib", version = "1.7.1")
bazel_dep(name = "platforms", version = "0.0.10")
bazel_dep(name = "toolchains_protoc", version = "0.3.7")
bazel_dep(name = "protobuf", version = "29.3")
bazel_dep(name = "rules_proto", version = "7.1.0")
bazel_dep(name = "gazelle", version = "0.42.0")
bazel_dep(name = "rules_oci", version = "2.0.1")
bazel_dep(name = "rules_pkg", version = "1.0.1")
bazel_dep(name = "rules_python", version = "0.37.1")
bazel_dep(name = "rules_python_gazelle_plugin", version = "0.37.1")
bazel_dep(name = "rules_uv", version = "0.14.0", dev_dependency = True)

host = use_extension("@aspect_bazel_lib//lib:extensions.bzl", "host", dev_dependency = True)
host.host()
use_repo(host, "aspect_bazel_lib_host")

host_platform = use_extension("@platforms//host:extension.bzl", "host_platform")
use_repo(host_platform, "host_platform")

####### PROTOBUF ##########
protoc = use_extension("@toolchains_protoc//protoc:extensions.bzl", "protoc")
protoc.toolchain(
    google_protobuf = "com_google_protobuf",
    version = "v28.0",
)
use_repo(protoc, "com_google_protobuf", "toolchains_protoc_hub")

register_toolchains("@toolchains_protoc_hub//:all")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.10",
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
use_repo(pip, "pypi")
use_repo(pip, "myapp_deps")
pip.parse(
    hub_name = "myapp_deps",
    python_version = "3.10",
    requirements_lock = "//myapp:requirements.txt",
)
pip.parse(
    hub_name = "pypi",
    python_version = "3.10",
    requirements_lock = "//myapp:requirements.txt",
)

oci = use_extension("@rules_oci//oci:extensions.bzl", "oci")

oci.pull(
    name = "distroless_base",
    digest = "sha256:ccaef5ee2f1850270d453fdf700a5392534f8d1a8ca2acda391fbb6a06b81c86",
    image = "gcr.io/distroless/base",
    platforms = [
        "linux/amd64",
    ],
)
oci.pull(
    name = "golang_base_oci",
    digest = "sha256:ad5c126b5cf501a8caef751a243bb717ec204ab1aa56dc41dc11be089fafcb4f",
    image = "index.docker.io/library/golang",
    platforms = [
        "linux/386",
        "linux/amd64",
        "linux/arm/v7",
        "linux/arm64/v8",
        "linux/mips64le",
        "linux/ppc64le",
        "linux/s390x",
        "unknown/unknown",
        "windows/amd64",
    ],
)
oci.pull(
    name = "python_base_oci",
    digest = "sha256:bc2e05bca883473050fc3b7c134c28ab822be73126ba1ce29517d9e8b7f3703b",
    image = "index.docker.io/library/python",
    platforms = [
        "linux/386",
        "linux/amd64",
        "linux/arm/v7",
        "linux/arm64/v8",
        "linux/mips64le",
        "linux/ppc64le",
        "linux/s390x",
        "unknown/unknown",
        "windows/amd64",
    ],
)
oci.pull(
    name = "golang_base_oci_arm64",
    image = "index.docker.io/library/golang",
    platforms = [
        "linux/arm64/v8",
    ],
    #   digest = "sha256:9e31fabdc95e5a5debd48e119d6da4670ae66b94a21581aed343e62f908b91ad",
    tag = "1.24.1",
)
oci.pull(
    name = "aws_provided_al2023_oci",
    digest = "sha256:4fa0ab8e5a5fe93f292fec8b1d71a1b1a7681a26d106e37a7891601a7ce217ce",
    image = "public.ecr.aws/lambda/provided",
    platforms = [
        "linux/amd64",
        "linux/arm64/v8",
    ],
)
oci.pull(
    name = "aws_provided_al2023_oci_arm64",
    image = "public.ecr.aws/lambda/provided",
    platforms = [
        "linux/arm64/v8",
    ],
    tag = "al2023-arm64",
)
oci.pull(
    name = "aws_lambda_python_oci",
    digest = "sha256:bed3a1eb8370b2ac5dd7cd431bfcad25ab3c70f03b4f1286fc7f7de850815239",
    image = "public.ecr.aws/lambda/python",
    platforms = [
        "linux/amd64",
        "linux/arm64/v8",
    ],
)
use_repo(oci, "aws_lambda_python_oci", "aws_provided_al2023_oci", "aws_provided_al2023_oci_arm64", "distroless_base", "golang_base_oci", "golang_base_oci_arm64", "python_base_oci")

[seberg]

This looks like you should talk to bazel specialists and not us. I would suggest you check what's inside the NumPy folder /myapp/myapp.runfiles/.myapp.venv/lib/python3.10/site-packages/numpy/_core. There should be some _multiarray_umath... file and the file extension will be wrong (e.g. incorrect architecture or python version).

That is something you have to figure out with the bazel setup.

[rgommers]

@KennyHoang-CS if you make the reproducing Bazel code as small and self-contained as possible, you may get an answer. Right now there is too much in there that makes it harder to work with your code, and it's not runnable as is (e.g., the Go stuff, the environment variables and the tar part of myapp are all irrelevant). An ideal reproducer is a small, self-contained git repository, so that the error can be reproduced after cloning on a machine that has Docker and Bazel installed.

I'll give you a guess for the root cause though: it looks like you're pulling a numpy wheel from PyPI, and Python from Amazon Linux which not be manylinux-compliant and hence not support that. I haven't checked recently, but Amazon Linux definitely used to not support wheels a while back.

[rgommers]

I'll close this issue, since it's not actionable for us in its current form. @KennyHoang-CS an update would still be appreciated, whether you solved it or have a more self-contained reproducer (I'd be happy to have a look at that).