Unexpected validation image on webdataset SDXL controlnet training script

[brycegoh]

Describe the bug

I am using the webdataset sdxl controlnet training script.

I ran the script two times with 2 different learning rates in which both runs gave unexpected generated validation images:

LR = 5e-5

Step 1050

Step 1100

LR = 4e-4

Step 50

Step 100

I understand this webdataset script differs from the example script in 2 ways:

1. The webdataset script uses the EDM formulation
2. The webdataset script also takes the image cropping top left coordinates into consideration when computing the embeddings whereas the example script takes a pre-defined crops_coords_top_left_h and crops_coords_top_left_w

Therefore, is the generated validation images expected as part of the training process?

Reproduction

These are my parameters:

! accelerate launch diffusers/examples/research_projects/controlnet/train_controlnet_webdataset.py \
--pretrained_model_name_or_path="stabilityai/stable-diffusion-xl-base-1.0" \
 --pretrained_vae_model_name_or_path="madebyollin/sdxl-vae-fp16-fix" \
 --output_dir="/workspace/output" \
 --train_shards_path_or_url="/workspace/data/dataset.tar" \
 --eval_shards_path_or_url="/workspace/data/valdiation_dataset.tar" \
 --control_type="seg" \
 --dataloader_num_workers=2 \
 --max_train_samples=20210 \
 --max_eval_samples=2000 \
 --mixed_precision="bf16" \
 --resolution=1024 \
 --learning_rate=5e-5 \
 --lr_scheduler="constant" \
 --max_train_steps=33690 \
 --validation_image "/workspace/segs/image1.png" \
 --validation_prompt "a living room with a red carpet and white walls"\
 --validation_steps=50 \
 --train_batch_size=6 \
 --gradient_accumulation_steps=1 \
 --report_to="wandb" \
 --tracker_project_name="sdxl-controlnet-run-02" \
 --seed=42 \
 --hub_model_id="helloHearthXHome/cn" \
 --checkpointing_steps=1685 \

Logs

No response

System Info

Diffusers v0.29.1

Who can help?

@sayakpaul @dn

[rphly]

also facing some issues, getting black images

[sayakpaul]

That script belongs to research_projects. So, we cannot guarantee its maintenance. So, I am going to convert this issue to a discussion and provide some suggestions I think might be worth trying out.