GPU not detected

[Kanss-code]

When am trying to run gradio app.py it reverts back to CPU even when am using L40.
Somehow your script is not recognising GPU

python -c "import torch; print(torch.__version__)"
2.2.0+cu118

python -c "import torch; print(torch.cuda.is_available())"
True

[Kanss-code]

And due to this revert back flash attention fails to intialize
Same issue has been posted here: #30

[iris2c]

Thank you for your comments. You may change inspiremusic/transformer/qwen_encoder.py QwenEmbeddingEncoder class into :
self.model = Qwen2ForCausalLM.from_pretrained(pretrain_path, device_map="cpu")

[Kanss-code]

Now getting this error
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

[Kanss-code]

And I have this confusion with this code changes, we are forcing the app to use CPU not GPU even tho we have installed all the dependencies nad packages.
Do we have any idea like why the gradio app is not recognizing GPU

[iris2c]

Note: If you're running this on Hugging Face Spaces, please set device_map="cpu" due to limited GPU availability.
If you're running the code locally with a GPU, you can keep the original setting device_map="auto".
To use FlashAttention, make sure it's installed first. Our latest environment supports flash-attn==2.7.4.post1.

[Kanss-code]

But when am activating conda env python 3.8 the latest flash attention shows a error stating :
ERROR: Package 'flash-attn' requires a different Python: 3.8.20 not in '>=3.9'

[GiambaDS86]

Thank you for your comments. You may change inspiremusic/transformer/qwen_encoder.py QwenEmbeddingEncoder class into : self.model = Qwen2ForCausalLM.from_pretrained(pretrain_path, device_map="cpu")

Even with this and multiple other fixes to run model explicitily on CPU (albeit my CUDA is detected by my env..) the generation breaks.

[Fluchw]

Oh, why am I doing the same on my 4090? It's so strange

[Fluchw]

Oh, why am I doing the same on my 4090? It's so strange哦，为什么我的 4090 也出现同样的情况？太奇怪了

I've solved it, just change the

model=InspireMusicModel (model_2="InspireMusic-1.5B-Long") 

to

model = InspireMusicModel(model_name = "InspireMusic-1.5B-Long",gpu=0)