What are the system requirenments and does it work on AMD GPUs due to LM Studio? What about Linux?

[gitchat1]

Thanks

[KungFuFurniture]

I just moments ago added to my LMStudio models, on an old AMD RX5700xt. Using vulkan.cpp on the GPU. WORKS FAST AND WELL!! Just downloaded in LMSTUDIO, cloned repo and ran orpheus-gguf.py.

Generated 167 audio segments
Generated 14.25 seconds of audio
Speech generation completed in 15.96 seconds

I will say, I cannot get a generation longer than 14 seconds. It always cuts off, when longer. But I have also only played with it for about 5 mins now.

I'm pretty stoked. This is dope.

Edit I opend gguf_orpheus.py and changed the default MAX_TOKENS to 4096 instead of 1200. No more 14 sec limit. I then added --max_tokens as an arg to my copy. But it might be a good idea, and pretty easy in an update. IMHO.

[rkfg]

It's possible to use this script with llama.cpp as well. I suspect any OpenAI-compatible API would do (which serves endpoints like /v1/completions and /v1/chat/completions). If you have llama.cpp compiled with CUDA and curl support, run the server like this: ./llama-server --hf-repo PkmX/orpheus-3b-0.1-ft-Q8_0-GGUF --hf-file orpheus-3b-0.1-ft-q8_0.gguf --port 1234 -ngl 256. It's a different 8-bit quant that I tried, if you want to use this author's repo, the command would be ./llama-server --hf-repo isaiahbjork/orpheus-3b-0.1-ft-Q4_K_M-GGUF --hf-file orpheus-3b-0.1-ft-q4_k_m.gguf --port 1234 -ngl 256 but I didn't test this case. The -ngl 256 parameter is excessive, it would load all layers to GPU this way. Omit it if you want to run it on CPU (slow, needs about 1.5 minutes, but works).

If you already have the model file downloaded, specify it with -m instead of --hf-repo/--hf-file.

[kkslidermight]

how did you get your sped that high? I'm using a 9070 following the 'Quick Setup' steps on the page with lmstudio.

Generated 4.95 seconds of audio
Speech generation completed in 17.17 seconds