Training ControlNet-Xs

[nguyendung1612]

Thank you @UmerHA for contributing the ControlNet demo. I tried to create a training script so I could train the model on my data but had some problems using the UNetControlNetXSModel.
Can you provide training or give me some suggestions?

[UmerHA]

What were the problems you ran into?

[nguyendung1612]

What were the problems you ran into?

I tried 2 ways:

1. I trained the UNetControlNetXSModel and saved the pretrained model. But when I checked your Pipeline and yours pretrain, I saw that you only saved the ControlNetXSAdapter class. How can you save ControlNetXSAdapter while training UNetControlNetXSModel?

2. I trained ControlNetXSAdapter class (optimizer and scheduler for this), but my loss did not change during training. I think it's because I only used UNetControlNetXSModel for forward without optimizing it.

[sayakpaul]

It should be turned into a discussion.