OOM with lora and half precision, but it works with full precision

[tshu-w]

I encountered an OOM error when using peft and half-precision, but training works fine with full precision. I believe there is an issue here, but I don't have time to complete the reproducible code. Therefore, I started this discussion for now.