Maintaining all the Control Net models with huge disc space is a really frustrating task. Now, forget about downloading and storing each type of ControlNet model one by one. Control Union for Stable Diffusion (SDXL) is the better alternative to the older ControlNets.
This model includes all types of control net models like Canny, Openpose, Depth, LineArt, Lineart, Mlsd, Scribble, Hed, Pidi (soft edge), Teed, Segment, Normal, etc. This is compatible with other open source Stable Diffusion XL, and LoRA models.
You can access the relevant research papers for in-depth understanding:
1. UniControl: A Unified Diffusion Model for Controllable Visual Generation.
2. Uni-ControlNet: All-in-One Control to Txt2Img Models.
3. Composer: Creative and Controllable Image Synthesis.
All the illustration for different ControlNets listed:
Installation in ComfyUI:
1. Install ComfyUI and get the basic ComfyUI knowledge.
2. Update ComfyUI by navigating to ComfyUI Manager and just select “Update ComfyUI“. This will avoid any future errors.
3. Next, download the ControlNet Union model for SDXL from the Hugging Face repository.
These are listed in the official repository-
(a) diffusion_pytorch_model (10 ControlNet included)
(b) diffusion_pytorch_model_promax (12 ControlNet included + 5 advanced editing)
Here, both are the same. The only difference between the two is that the latter includes “Tile Super“, “Tile Deblur“, “Inpaint” etc.
Choose any one of them as per your requirements. Save it inside the “ComfyUI/models/controlnet” folder.
4. After downloading, rename it to some unique name to avoid any mismatches and errors in the future. We renamed it to “ControlNet-Union-SDXL“. Make sure to use Stable Diffusion XL (SDXL) checkpoints only.
5. Now, restart ComfyUI and click the refresh button to take effect.
Many people are facing with “ControlNet Preprocessor” list unavailable error for not populating in the dropdown list and just with “tile” listed. The solution is to loading the ControlNet Preprocessor node, in the preprocessor selection item. You need to install custom nodes “ComfyUI’s ControlNet Auxiliary Preprocessors” from ComfyUI Manager .
Installation in Automatic1111/ForgeUI:
1. You need Automatic1111 /ForgeUI installed and upgrade your ControlNet extension from the “Extensions” tab.
2. Just download the model from the official Hugging Face repository.
These models listed-
(a) diffusion_pytorch_model (10 ControlNet included)
(b) diffusion_pytorch_model_promax (12 ControlNet included + 5 advanced editing)
The only difference between the above listed is that the latter includes “Tile Super“, “Tile Dblur“, “Inpaint” etc.
After downloading, save it inside the “stable-diffusion-webui/models/ControlNet” folder, as usual, as you did with older ControlNet models. It is recommended to rename it to “controlnet++_union_sdxl” to avoid any errors.
3. Restart your Webui by clicking on “Apply and RestartUI“. You will get a new section like the earlier ControlNets.
Choose your Stable Diffusion XL checkpoints. Upload your image. Move into the ControlNet section and in the “Model” section, and select “controlnet++_union_sdxl” from the dropdown menu. Again select the “Preprocessor” you want like canny, soft edge, etc.
The Controlnet Union is new, and currently some ControlNet models are not working as per your desired results. So, you need to wait more to get it efficiently working.
Conclusion:
The all-in-one control net model is the problem solver over the traditional ControlNet. It is now compatible with both Stable Diffusion WebUIs. Apart from this, we have extra advanced editing features included.
