Questions of pre-training LoRA with other modules simutaneously #8472
Description
Describe the bug
Sorry for asking questions unrelated to diffusers... I am trying to jountly train an ELLA model with a UNET LoRA (rank: 64). However, I confronted a weird situation, the ELLA model worked well while the LoRA crashed and made the output like random noise. Have you guys witnessed this phenomenon?
Reproduction
I have tried multiple ways, like FP16, FP32, Deepspeed, and checked my code many times. It confused me for a long time. I would appreciate if the community conld provide some suggestions or guidiances.
Logs
No response
System Info
diffusersversion: 0.28.0.dev0- Platform: Linux-6.5.0-35-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- Huggingface_hub version: 0.21.3
- Transformers version: 4.30.2
- Accelerate version: 0.21.0
- xFormers version: 0.0.24
- Using GPU in script?:
- Using distributed or parallel set-up in script?:
Who can help?
No response