Train sd3 controlnet but result is bad

[xduzhangjiayu]

Describe the bug

Hi,
I tried to train sd3 controlnet with examples\controlnet\train_controlnet_sd3.py, but it seems the validation results are really bad, and the training loss was oscillating all the time, it really confused me for a long time, could you please give some advice? Thanks a lot !

Here are some validation results and train loss:
validation output on steps 2900:

loss function in tensorboard

Reproduction

accelerate launch examples/controlnet/train_controlnet_sd3.py --pretrained_model_name_or_path=stable-diffusion-3-medium-diffusers --output_dir=output --resolution=1024 --learning_rate=1e-5 --validation_image "./conditioning_image_1.png" "./conditioning_image_2.png" --validation_prompt "red circle with blue background" "cyan circle with brown floral background" --train_batch_size=1 --gradient_checkpointing --gradient_accumulation_steps=4

Logs

No response

System Info

diffusers: 0.31.0.dev (latest)
torch: 2.4.0

Who can help?

@sayakpaul @DN6 @asomoza

[sayakpaul]

This should belong to discussions. Ccing @PromeAIpro who contributed the script.