RuntimeError: Found empty `transformer-engine` meta package installed

[GeneralSubhra]

code:
import modal

Use the NGC PyTorch container which has TE and Torch pre-linked

24.07 matches CUDA 12.4/12.5 environments well

evo2_image = (
modal.Image.from_registry(
"nvcr.io/nvidia/pytorch:24.10-py3",
add_python="3.12"
)
.apt_install([
"build-essential",
"cmake",
"ninja-build",
"git",
"libcudnn9-dev-cuda-12",
])
.env({"CXX": "/usr/bin/g++"})
.run_commands("pip install wheel setuptools packaging")
.run_commands(
"pip install packaging",
"git clone https://github.com/arcinstitute/evo2 && cd evo2 && pip install -e .",
"pip install flash-attn --no-build-isolation"
)
.pip_install_from_requirements("requirements.txt")
)

error:

NVIDIA Release 24.10 (build 114410972)
PyTorch Version 2.5.0a0+e000cf0
Container image Copyright (c) 2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Copyright (c) 2014-2024 Facebook Inc.
Copyright (c) 2011-2014 Idiap Research Institute (Ronan Collobert)
Copyright (c) 2012-2014 Deepmind Technologies (Koray Kavukcuoglu)
Copyright (c) 2011-2012 NEC Laboratories America (Koray Kavukcuoglu)
Copyright (c) 2011-2013 NYU (Clement Farabet)
Copyright (c) 2006-2010 NEC Laboratories America (Ronan Collobert, Leon Bottou, Iain Melvin, Jason Weston)
Copyright (c) 2006 Idiap Research Institute (Samy Bengio)
Copyright (c) 2001-2004 Idiap Research Institute (Ronan Collobert, Samy Bengio, Johnny Mariethoz)
Copyright (c) 2015 Google Inc.
Copyright (c) 2015 Yangqing Jia
Copyright (c) 2013-2016 The Caffe contributors
All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

Traceback (most recent call last):
File "/pkg/modal/_runtime/container_io_manager.py", line 947, in handle_input_exception
yield
File "/pkg/modal/_container_entrypoint.py", line 171, in run_input_sync
values = io_context.call_function_sync()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/pkg/modal/_runtime/container_io_manager.py", line 225, in call_function_sync
expected_value_or_values = self.finalized_function.callable(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/main.py", line 47, in run_brca1_analysis
from evo2 import Evo2
File "/workspace/evo2/evo2/init.py", line 1, in
from .models import Evo2
File "/workspace/evo2/evo2/models.py", line 12, in
from vortex.model.model import StripedHyena
File "/usr/local/lib/python3.12/site-packages/vortex/model/model.py", line 15, in
from vortex.model.layers import (
File "/usr/local/lib/python3.12/site-packages/vortex/model/layers.py", line 10, in
from transformer_engine.pytorch import Linear
File "/usr/local/lib/python3.12/site-packages/transformer_engine/init.py", line 11, in
import transformer_engine.common
File "/usr/local/lib/python3.12/site-packages/transformer_engine/common/init.py", line 386, in
sanity_checks_for_pypi_installation()
File "/usr/local/lib/python3.12/site-packages/transformer_engine/common/init.py", line 215, in sanity_checks_for_pypi_installation
raise RuntimeError(
RuntimeError: Found empty transformer-engine meta package installed. Install transformer-engine with framework extensions via'pip3 install --no-build-isolation transformer-engine[pytorch,jax]==VERSION' or 'pip3 install transformer-engine[core]for the TE core lib only. Thecore_cu12orcore_cu13extra deps can be used to specify CUDA version for the TE core lib. Stopping app - uncaught exception raised locally: RuntimeError("Found emptytransformer-enginemeta package installed. Installtransformer-engine with framework extensions via'pip3 install --no-build-isolation transformer-engine[pytorch,jax]==VERSION' or 'pip3 install transformer-engine[core] for the TE core lib only. The core_cu12 or core_cu13 extra deps can be used to specify CUDA version for the TE core lib.").
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ D:\variant-analysis-evo2\main.py:171 in main │
│ │
│ 170 def main(): │
│ ❱ 171 │ brca1_example.local() │
│ 172 │
│ │
│ C:\Users\Subhranil\AppData\Local\Programs\Python\Python312\Lib\site-packages\modal_functions.py │
│ :1835 in local │
│ │
│ 1834 │ │ │ fun = info.raw_f │
│ ❱ 1835 │ │ │ return fun(*args, **kwargs) │
│ 1836 │ │ else: │
│ │
│ D:\variant-analysis-evo2\main.py:166 in brca1_example │
│ │
│ 165 │ #Run inference │
│ ❱ 166 │ returns = run_brca1_analysis.remote() │
│ 167 │
│ │
│ C:\Users\Subhranil\AppData\Local\Programs\Python\Python312\Lib\site-packages\modal_object.py:34 │
│ 0 in wrapped │
│ │
│ 339 │ │ await self.hydrate() │
│ ❱ 340 │ │ return await method(self, *args, **kwargs) │
│ 341 │
│ │
│ C:\Users\Subhranil\AppData\Local\Programs\Python\Python312\Lib\site-packages\modal_functions.py │
│ :1766 in remote │
│ │
│ 1765 │ │ │
│ ❱ 1766 │ │ return await self._call_function(args, kwargs) │
│ 1767 │
│ │
│ C:\Users\Subhranil\AppData\Local\Programs\Python\Python312\Lib\site-packages\modal_functions.py │
│ :1710 in _call_function │
│ │
│ 1709 │ │ │
│ ❱ 1710 │ │ return await invocation.run_function() │
│ 1711 │
│ │
│ C:\Users\Subhranil\AppData\Local\Programs\Python\Python312\Lib\site-packages\modal_functions.py │
│ :293 in run_function │
│ │
│ 292 │ │ │ item = await self._get_single_output() │
│ ❱ 293 │ │ │ return await _process_result(item.result, item.data_format, self.stub, self. │
│ 294 │
│ │
│ C:\Users\Subhranil\AppData\Local\Programs\Python\Python312\Lib\site-packages\modal_utils\functi │
│ on_utils.py:529 in _process_result │
│ │
│ 528 │ │ │ │
│ ❱ 529 │ │ │ raise exc_with_hints(exc) │
│ 530 │
│ │
│ ...Remote call to Modal Function (ta-01KDNHJ6KW2FT0412E08FHQAF0)... │
│ │
│ /root/main.py:47 in run_brca1_analysis │
│ │
│ ❱ 47 from evo2 import Evo2 │
│ │
│ │
│ /workspace/evo2/evo2/init.py:1 in │
│ │
│ ❱ 1 from .models import Evo2 │
│ │
│ │
│ /workspace/evo2/evo2/models.py:12 in │
│ │
│ ❱ 12 from vortex.model.model import StripedHyena │
│ │
│ │
│ /usr/local/lib/python3.12/site-packages/vortex/model/model.py:15 in │
│ │
│ ❱ 15 from vortex.model.layers import ( │
│ │
│ │
│ /usr/local/lib/python3.12/site-packages/vortex/model/layers.py:10 in │
│ │
│ ❱ 10 from transformer_engine.pytorch import Linear │
│ │
│ │
│ /usr/local/lib/python3.12/site-packages/transformer_engine/init.py:11 in │
│ │
│ ❱ 11 import transformer_engine.common │
│ │
│ │
│ /usr/local/lib/python3.12/site-packages/transformer_engine/common/init.py:386 in │
│ │
│ ❱ 386 sanity_checks_for_pypi_installation() │
│ │
│ │
│ /usr/local/lib/python3.12/site-packages/transformer_engine/common/init.py:215 in │
│ sanity_checks_for_pypi_installation │
│ │
│ ❱ 215 raise RuntimeError( │
│ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Found empty transformer-engine meta package installed. Install transformer-engine with framework extensions
via'pip3 install --no-build-isolation transformer-engine[pytorch,jax]==VERSION' or 'pip3 install transformer-engine[core] for the TE core lib only. Thecore_cu12orcore_cu13` extra deps can be used to specify CUDA version for the TE core lib.

[schmigle]

I have this issue as well, has there been any progress?

[mdanner93]

Using the NGC PyTorch container with TE and Flash-Attention pre-installed and installing evo2 and vertex with no-deps argument did the trick for me:

# 24.12 has Python 3.12, PyTorch 2.5, CUDA 12.6, Transformer Engine, Flash Attention
FROM nvcr.io/nvidia/pytorch:24.12-py3

# Set environment variables
ENV DEBIAN_FRONTEND=noninteractive
ENV MAX_JOBS=8

# Install ninja first for faster builds
RUN pip install --no-cache-dir ninja packaging

# Uninstall NGC's flash-attn and reinstall compatible version
RUN pip uninstall -y flash-attn || true && \\
    pip install --no-cache-dir flash-attn==2.7.3 --no-build-isolation

In my case evo2/vertex seemed to overwrite the working transformer-engine with an empty meta package.

Edit: It seems that the flash-attn coming with the base image is not compatible. -> flash-attn==2.7.3 seems to work.