Why promp processing is so slow with AOCL-BLAS?

[paat]

I was trying to performance tune my CPU inference speed using AOCL, but it backfired. Did I do something wrong?
CPU: Threadripper PRO 7965WX

AOCL

Cmake params

    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_NATIVE=ON \
    -DGGML_BLAS=ON \
    -DGGML_BLAS_VENDOR=AOCL \
    -DBLAS_LIBRARIES=/opt/AMD/aocl/aocl-linux-gcc-5.1.0/gcc/lib/libblis.so \
    -DBLAS_INCLUDE_DIRS=/opt/AMD/aocl/aocl-linux-gcc-5.1.0/gcc/include \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES=120 \
    -DGGML_CUDA_F16=ON \
    -DGGML_CUDA_FA=ON \
    -DGGML_CUDA_GRAPHS=ON \
    -DGGML_LTO=ON \
    -DBUILD_SHARED_LIBS=OFF \
    -DLLAMA_BUILD_EXAMPLES=ON \
    -DLLAMA_BUILD_TESTS=OFF \
    -DLLAMA_BUILD_SERVER=ON \
    -DLLAMA_BUILD_TOOLS=ON \
    -DLLAMA_CURL=ON \
    -DGGML_OPENMP=ON \
    -DGGML_CUDA_FA_ALL_QUANTS=ON

llama-bench result

llama.cpp/build/bin$ ./llama-bench -m /mnt/data/ai/models/llm/Mistral-7B-Instruct-v0.3.Q5_K_M.gguf -t 24 -ngl 0 --progress
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| llama 7B Q5_K - Medium         |   4.78 GiB |     7.25 B | CUDA,BLAS  |      24 |           pp512 |          8.66 ± 0.31 |
| llama 7B Q5_K - Medium         |   4.78 GiB |     7.25 B | CUDA,BLAS  |      24 |           tg128 |         31.43 ± 0.06 |

OpenBLAS

Cmake params

    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_NATIVE=ON \
    -DGGML_BLAS=ON \
    -DGGML_BLAS_VENDOR=OpenBLAS \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES=120 \
    -DGGML_CUDA_F16=ON \
    -DGGML_CUDA_FA=ON \
    -DGGML_CUDA_GRAPHS=ON \
    -DGGML_LTO=ON \
    -DBUILD_SHARED_LIBS=OFF \
    -DLLAMA_BUILD_EXAMPLES=ON \
    -DLLAMA_BUILD_TESTS=OFF \
    -DLLAMA_BUILD_SERVER=ON \
    -DLLAMA_BUILD_TOOLS=ON \
    -DLLAMA_CURL=ON \
    -DGGML_OPENMP=ON \
    -DGGML_CUDA_FA_ALL_QUANTS=ON

llama-bench result

llama.cpp/build/bin$ ./llama-bench -m /mnt/data/ai/models/llm/Mistral-7B-Instruct-v0.3.Q5_K_M.gguf -t 24 -ngl 0 --progress
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| llama 7B Q5_K - Medium         |   4.78 GiB |     7.25 B | CUDA,BLAS  |      24 |           pp512 |         75.14 ± 0.65 |
| llama 7B Q5_K - Medium         |   4.78 GiB |     7.25 B | CUDA,BLAS  |      24 |           tg128 |         29.77 ± 0.04 |

[paat]

found out the problem. I should have used multithread library:
-DBLAS_LIBRARIES=/opt/AMD/aocl/aocl-linux-gcc-5.1.0/gcc/lib/libblis-mt.a \

[paat]

Note, that you need to set OMP_NUM_THREADS=24 enviroment variable for it to work properly

i see about 7-8% pp performance improvement compared to OpenBlas

llama.cpp/build/bin$ OMP_NUM_THREADS=24 ./llama-bench -m /mnt/data/ai/models/llm/Mistral-7B-Instruct-v0.3.Q5_K_M.gguf -ngl 0 -t 24
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
| model                          |       size |     params | backend    | threads |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| llama 7B Q5_K - Medium         |   4.78 GiB |     7.25 B | CUDA,BLAS  |      24 |           pp512 |         76.34 ± 0.64 |
| llama 7B Q5_K - Medium         |   4.78 GiB |     7.25 B | CUDA,BLAS  |      24 |           tg128 |         31.45 ± 0.04 |

build: aa79524c (6027)