Rocm support planned

[DominicBoettger]

As I love using Radeons under Linux I would happy to see rocm support.
Do you plan to support rocm/AMD in the future?

[marcussacana]

I want it too, this project is a light for the overpriced GPUs.

[BatSmacker84]

I am using a Radeon RX 7800XT under Linux to run exo and (from my novice perspective) it appears like exo is properly utilizing my GPU. I have no idea if this is "proper" ROCm support on exo's part or if it is running through a translation layer of some kind. Clarification by the main developer on the matter would be nice, as well as updated documentation on how/if exo supports ROCm/AMD GPUs and how to install and run exo with an AMD GPU.

If this issue is about exo not properly recognizing AMD GPU info (ie. name/model and performance), then PR #417 is my attempt at allowing exo to gather capability info for AMD GPUs. However, this is entirely cosmetic because even without that PR my GPU is utilized by exo.

Not trying to dismiss the issue, I just think a little clarification on what the issue is would be helpful to solve it.

[krenax]

Any plans for ROCM support?

[zhang3989]

how about AMD device？

[jasoncouture]

7900 XTX here, cheaper than Nvidia cards, and consumer class 24GB of ram. Inference engines that support it have amazing performance.

So, +1 here.

[deftdawg]

7900 XTX here

Have you tried: AMD=1 exo
Tinygrad (which exo uses on x86), should support the 7900 XTX without RoCM because tinycorp wrote their own stack for tinybox red.

[deftdawg]

Got back to messing with this, the AMD code isn't quite working...

It tries to use pyrsmi to detect the AMD gpu (assuming AMD=1 is set in the env), but then sets the GPU name to "{gpu_name}" and the flops to 0... 😞

When I run pyrsmi on my system, it only detects the GPU when not executed in exo (maybe a Nix-problem relating to /usr/share/libdrm/amdgpu.ids not existing off of /), but when I run the example code it gives me a hex value for the device name... That means even if exo were trying to lookup the flops in the CHIP_FLOPS table, it wouldn't find anything ...

python3 device_info.py 
no. of devices = 1
 
device id       device name     total memory(GB)  used memory(MB)
-----------------------------------------------------------------
     0    0x1002                17.16           5625.43

So I rip out pyrsmi and put back pyamdgpuinfo lookup from @BatSmacker84 's patch and we can get the device properly detected again... yay!

Download a model, but then I bomb out with a tinygrad error:

Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): No module named 'tinygrad.runtime.autogen.am'

Getting closer, but still have some dependency work to do....

Here's the latest patch that fixes detection:

diff --git a/exo/topology/device_capabilities.py b/exo/topology/device_capabilities.py
index ae28e75..e60003f 100644
--- a/exo/topology/device_capabilities.py
+++ b/exo/topology/device_capabilities.py
@@ -198,22 +198,19 @@ async def linux_device_capabilities() -> DeviceCapabilities:
       flops=CHIP_FLOPS.get(gpu_name, DeviceFlops(fp32=0, fp16=0, int8=0)),
     )
   elif Device.DEFAULT == "AMD":
-    # For AMD GPUs, pyrsmi is the way (Official python package for rocm-smi)
-    from pyrsmi import rocml
+    import pyamdgpuinfo

-    rocml.smi_initialize()
-    gpu_name = rocml.smi_get_device_name(0).upper()
-    gpu_memory_info = rocml.smi_get_device_memory_total(0)                                                                                                      +    gpu_raw_info = pyamdgpuinfo.get_gpu(0)
+    gpu_name = gpu_raw_info.name
+    gpu_memory_info = gpu_raw_info.memory_info["vram_size"]

     if DEBUG >= 2: print(f"AMD device {gpu_name=} {gpu_memory_info=}")

-    rocml.smi_shutdown()
-
     return DeviceCapabilities(
-      model="Linux Box ({gpu_name})",
+      model="Linux Box (" + gpu_name + ")",
       chip=gpu_name,
       memory=gpu_memory_info // 2**20,
-      flops=DeviceFlops(fp32=0, fp16=0, int8=0),
+      flops=CHIP_FLOPS.get(gpu_name, DeviceFlops(fp32=0, fp16=0, int8=0)),
     )

   else:
diff --git a/setup.py b/setup.py
index de242f5..421421e 100644
--- a/setup.py
+++ b/setup.py
@@ -20,6 +20,7 @@ install_requires = [
   "prometheus-client==0.20.0",
   "protobuf==5.28.1",
   "psutil==6.0.0",
+  "pyamdgpuinfo==2.1.6;platform_system=='Linux'",
   "pydantic==2.9.2",
   "requests==2.32.3",
   "rich==13.7.1",

[deftdawg]

Yay! GPU inference on my 6900XT... nimlgen from Tinycorp was able to fix Tinygrad's ctl stack size for RX 6000 / gfx1030 cards

That combined with my exo modifications (above) has everything working for me now. Will post a PR here later tonight.

[deftdawg]

One thing I noticed today was that there's still a reference to pyrsmi in the Windows section of device_capabilities class... So while this is closed for Linux, I expect it won't work as is on Windows. I don't have any Windows devices, but if you're on Windows and it doesn't work perhaps give it a go under WSL.

[FlorianHeigl]

@deftdawg just FYI I did a git pull, upgraded python, reran install.sh earlier today and am still seeing the runtime error message running $ AMD=1 /home/me/.local/bin/exo:

Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): No module named 'tinygrad.runtime.autogen.am'

The detection seems to happen, in part, so it sees the AMD card but at 0 tops.
Linux Box (AMD R/deon Pro VII) 15GB

I can remember there's some little database file that has the performance data, and i think that is not related to the issue.
(i'll definitely need to look into that rounding error, it stings)

I see your patches were merged so I think there's 3 possibilites

• I messed up (most likely though your patches must be in effect since tinygrad no longer complains about the GPU model per se)
• Something is missing from the upstreaming
• Card specific issue since it's older (for this i got rocm 6.1.3 & ubuntu 22.04 so it should be ok?)

In any case thanks for your investigations, seems to be many steps further to target now.

[deftdawg]

Failed to fetch completions: Error processing prompt (see logs with DEBUG>=2): No module named 'tinygrad.runtime.autogen.am'

This is a tinygrad / ROCm problem, best to start with tinygrad and make sure you can run the simple tensor example on their github page.

The detection seems to happen, in part, so it sees the AMD card but at 0 tops. Linux Box (AMD R/deon Pro VII) 15GB

There's no entry for that card in CHIP_FLOPS so it will show 0 tops.

Best bet is to use the official docker container pass the GPU device in (that rules out ROCm), then run tinygrad inside it, and then run exo.

This is how I test on NixOS where ROCm is still being upgraded to 6.3....

sudo systemctl stop firewall # disable firewall if using remotely
docker run -it \
    --device=/dev/kfd --device=/dev/dri \
    -e AMD=1 -e DEBUG=2
	-p 52415:52415 \
--name tinytest rocm/dev-ubuntu-22.04

[alvaro-gh]

My two cents, the error for the missing module was fixed on this commit:

• tinygrad/tinygrad@3e753fd

So the setup.py on the exo project should be updated given that the tinygrad dependency is tied to an older commit. I'll test this tomorrow on my own and get back (looking at this on my phone right now).