Unable to build docker image on Apple M4 chip (required for development, not for real inference)

[shkarupa-alex]

System Info

This solution from README doesn't work

DISCLAIMER
As explained here pytorch/pytorch#81224, Metal / MPS is not supported via Docker. As such inference will be CPU bound and most likely pretty slow when using this docker image on an M1/M2 ARM CPU.

docker build . -f Dockerfile --platform=linux/arm64

It raises error:

9.351 E: Unable to locate package intel-oneapi-mkl-devel
------
Dockerfile:35
--------------------
  34 |     
  35 | >>> RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
  36 | >>>     intel-oneapi-mkl-devel=2024.0.0-49656 \
  37 | >>>     build-essential \
  38 | >>>     && rm -rf /var/lib/apt/lists/*
  39 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends     intel-oneapi-mkl-devel=2024.0.0-49656     build-essential     && rm -rf /var/lib/apt/lists/*" did not complete successfully: exit code: 100

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

1. Obtain Apple with M4 chip (MacBook Pro)
2. Checkout v1.7.0 branch
3. Run docker build . -f Dockerfile --platform=linux/arm64

Expected behavior

There should be way to use this project as a part of docker compose (for development).

[polarathene]

Unable to locate package intel-oneapi-mkl-devel

That dependency was added without much context in Nov 2024 via this PR. There's a separate Intel optimized Dockerfile that is implemented a little differently, so it's a bit unclear why that was brought into a generic CPU image 🤷‍♂

Presumably the package is no longer available with the version pinned. You could try editing the Dockerfile to remove the =2024.0.0-49656 portion and see if that works.

Version pinning deps can be important, but in doing so for that to work reliably over time other deps including the base image should be pinned with care taken when updating the image (or just building at a later date depending on any deps not version pinned).

---

Oh right... that dep is probably x86_64 arch only for Intel, so you'd probably need to avoid that entirely (like the Dockerfile prior to that PR, or compare to the current Dockerfile and remove the changes the PR introduced).

[alvarobartt]

Hey @shkarupa-alex, as you point out, MPS over Docker won't work until supported if that eventually happens, in the meantime, you should be able to leverage CPU-only inference after building the image with the --platform=linux/amd64 (instead of --platform=linux/arm64) with the command docker build --platform=linux/amd64 -f Dockerfile .. That basically means that instead of leveraging the native MPS architecture (ARM64), it will leverage the AMD64 architecture emulating x86_64 instructions allowing you to run Intel/AMD images on your Apple Silicon device.

Alternatively, you mentioned that you are planning to use that for development purposes only, then maybe the recommended alternative would be to go via the Cargo path and just install the Cargo dependencies and run the binary instead to be able to leverage MPS inference locally with the command cargo install --path router -F metal.

@Narsil should we update the README.md at https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#apple-m1m2-arm64-architectures to use --platform=linux/amd64 instead? Or is that something that needs to be fixed within the Dockerfile? Guessing it should be just the README.md, but please let me know otherwise.

Thanks in advance 🤗

[polarathene]

Or is that something that needs to be fixed within the Dockerfile?

It should be fixed in the Dockerfile. Only install x86_64 target specific deps when building for that target. If you need guidance for that I can provide it.