Training ControlNet on Stable Diffusion Inpaint

[engrmusawarali]

How do we create weights for the controlnet inpaint which uses stable diffusion inpaint as backbone @patrickvonplaten @sayakpaul @dhruv Nair

[patrickvonplaten]

Hey @engrmusawarali,

Do you mean how to fine-tune a stable diffusion base checkpoint for controlnet inpainting?

[sayakpaul]

Seems fine. What's the problem with that?

[engrmusawarali]

@sayakpaul i prepared this using suggestion from instruct pix2pix, I wanted to confirm If this is correct way, It would be good if this option is integrated in the pipelines. Thanks a lot for hint

[sayakpaul]

If we have good checkpoints and convincing results, then I don't see any reason not to.

[engrmusawarali]

@sayakpaul, it is totally dependent on the problem we are solving, my experience is really good, with controlnet inpaint with stable diffusion inpaint, before this I was initializing cntrolnet unet with stable diffusion base unet, But i wanted to follow the exact protocols proposed by controlnets.

[shoaibmustafakhan]

I'm working a research project myself and I would love to know if you've developed a finetuning script for the model?