[Bug]: controlnet got very slow after today's nvidia update

[alenknight]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

i saw something similar in a few forums but not sure if these are from today. after nvidia's latest patch update, controlnet got very very SLOW..... and Tile using upscaling is almost unusable.

Steps to reproduce the problem

1. Go to .... render image
2. run controlnet tile with any image using tile diffusion or ultimate sd upscale
3. render

What should have happened?

on a RTX 3090... i'm getting about 15s/t for images to render... when i was getting prior about 1-2s/t for large images.

Commit where the problem happens

webui: AUTOMATIC1111/stable-diffusion-webui@394ffa7
controlnet: 8e143d3

What browsers do you use to access the UI ?

Mozilla Firefox, Google Chrome, Brave, Apple Safari, Microsoft Edge

Command Line Arguments

--listen --port 7777 --opt-split-attention --disable-nan-check --api --enable-insecure-extension-access

List of enabled extensions

Console logs

venv "C:\AI\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: v1.4.0
Commit hash: 394ffa7b0a7fff3ec484bcd084e673a8b301ccc8
Installing requirements



current transparent-background 1.2.4

Requirement already satisfied: send2trash~=1.8 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (1.8.2)
Requirement already satisfied: dynamicprompts[attentiongrabber,magicprompt]~=0.27.0 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (0.27.0)
Requirement already satisfied: jinja2~=3.1 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.1.2)
Requirement already satisfied: pyparsing~=3.0 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.1.0)
Requirement already satisfied: transformers[torch]~=4.19 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (4.25.1)
Requirement already satisfied: MarkupSafe>=2.0 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from jinja2~=3.1->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2.1.3)
Requirement already satisfied: huggingface-hub<1.0,>=0.10.0 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (0.15.1)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (0.13.3)
Requirement already satisfied: numpy>=1.17 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (1.23.5)
Requirement already satisfied: filelock in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.12.2)
Requirement already satisfied: regex!=2019.12.17 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2023.6.3)
Requirement already satisfied: pyyaml>=5.1 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (6.0)
Requirement already satisfied: requests in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2.31.0)
Requirement already satisfied: packaging>=20.0 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (23.1)
Requirement already satisfied: tqdm>=4.27 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (4.65.0)
Requirement already satisfied: torch!=1.12.0,>=1.7 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2.0.1+cu118)
Requirement already satisfied: fsspec in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from huggingface-hub<1.0,>=0.10.0->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2023.6.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from huggingface-hub<1.0,>=0.10.0->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (4.6.3)
Requirement already satisfied: networkx in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from torch!=1.12.0,>=1.7->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.1)
Requirement already satisfied: sympy in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from torch!=1.12.0,>=1.7->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (1.12)
Requirement already satisfied: colorama in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from tqdm>=4.27->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (0.4.6)
Requirement already satisfied: idna<4,>=2.5 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from requests->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from requests->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (1.26.16)
Requirement already satisfied: charset-normalizer<4,>=2 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from requests->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (3.1.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from requests->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (2023.5.7)
Requirement already satisfied: mpmath>=0.19 in c:\ai\stable-diffusion-webui\venv\lib\site-packages (from sympy->torch!=1.12.0,>=1.7->transformers[torch]~=4.19->dynamicprompts[attentiongrabber,magicprompt]~=0.27.0) (1.3.0)
sd-dynamic-prompts installer: running 'C:\AI\stable-diffusion-webui\venv\Scripts\python.exe' -m pip install 'send2trash~=1.8' 'dynamicprompts[attentiongrabber,magicprompt]~=0.27.0'


Installing opencv-python
Installing opencv-python for pixel extension



Checking roop requirements
Install insightface==0.7.3
Installing sd-webui-roop requirement: insightface==0.7.3
Install onnx==1.14.0
Installing sd-webui-roop requirement: onnx==1.14.0
Install onnxruntime==1.15.0
Installing sd-webui-roop requirement: onnxruntime==1.15.0
Install opencv-python==4.7.0.72
Installing sd-webui-roop requirement: opencv-python==4.7.0.72

Checking roop requirements
Install insightface==0.7.3
Installing sd-webui-roop requirement: insightface==0.7.3
Install onnx==1.14.0
Installing sd-webui-roop requirement: onnx==1.14.0
Install onnxruntime==1.15.0
Installing sd-webui-roop requirement: onnxruntime==1.15.0
Install opencv-python==4.7.0.72
Installing sd-webui-roop requirement: opencv-python==4.7.0.72

If submitting an issue on github, please provide the full startup log for debugging purposes.

Initializing Dreambooth
Dreambooth revision: dc413a14379b165355502d9f65856c40a4bb5b6f
Successfully installed accelerate-0.19.0 fastapi-0.94.1 gitpython-3.1.31 transformers-4.29.2


Does your project take forever to startup?
Repetitive dependency installation may be the reason.
Automatic1111's base project sets strict requirements on outdated dependencies.
If an extension is using a newer version, the dependency is uninstalled and reinstalled twice every startup.

[+] xformers version 0.0.17 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.19.0 installed.
[+] diffusers version 0.16.1 installed.
[+] transformers version 4.29.2 installed.
[+] bitsandbytes version 0.35.4 installed.


Launching Web UI with arguments: --listen --port 7777 --opt-split-attention --disable-nan-check --api --enable-insecure-extension-access
No module 'xformers'. Proceeding without it.
[-] ADetailer initialized. version: 23.6.4, num models: 11
2023-06-29 22:08:47,566 - ControlNet - INFO - ControlNet v1.1.227
ControlNet preprocessor location: C:\AI\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads
2023-06-29 22:08:48,058 - ControlNet - INFO - ControlNet v1.1.227
2023-06-29 22:08:53,796 - roop - INFO - roop v0.0.2
2023-06-29 22:08:53,798 - roop - INFO - roop v0.0.2
2023-06-29 22:08:53,801 - roop - INFO - roop v0.0.2
Loading weights [980cb713af] from C:\AI\stable-diffusion-webui\models\Stable-diffusion\realistic\reliberate_v10.safetensors
Creating model from config: C:\AI\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Textual inversion embeddings loaded(121): advntr, angry512, anime512, ansm1th_ananicolesmith, AS-Adult, AS-Adult-neg, AS-Elderly, AS-MidAged, AS-Young, AS-Young-neg, AS-Younger, AS-YoungerV2, AS-Youngest, AS-YoungestV2, AS-YoungV2, AS-YoungV2-neg, Asian-Less, Asian-Less-Neg, Asian-Less-Toon, AuroraNegative, awful512, bad-artist, bad-artist-anime, bad-hands-5, bad-image-9600, bad-image-v2-11000, bad-image-v2-27000, bad-image-v2-39000, bad-picture-chill-1v, bad-picture-chill-32v, bad-picture-chill-75v, bad_prompt, bad_prompt_version2, bad_prompt_version2-neg, BadDream, badhandsv5-neg, badhandv4, bazaldua_v2-10000, beauty512, boring_e621, corneo_covering_breasts_two_hands, defiance512, dichen, easynegative, emb-anada, emb-avril, emb-rrf2, emmastone, EmWat69, FastNegativeEmbedding, FastNegativeEmbeddingStrong, fcDetailPortrait, fcHeatPortrait, fcNeg-neg, fcPortrait, grin512, GS-Boyish, GS-DeFeminize-neg, GS-DeMasculate-neg, GS-Masculine, happy512, HAROQUE, HRGiger, HRGigerEmb, is, JuggernautNegative-neg, kaleidcp-3450, KHFB, laugh512, linds3yp3las, lnrneg, lnrpos, lr, MLKEmb, n3t0p, neg_anime512, neg_facelift512, neg_realism512, negative_hand-neg, nervous512, ng_deepnegative_v1_16t, ng_deepnegative_v1_2t, ng_deepnegative_v1_32t, ng_deepnegative_v1_4t, ng_deepnegative_v1_64t, ng_deepnegative_v1_75t, nRealisticVision, opt-6000, Portrait512, pureerosface_v1, realisticvision-negative-embedding, render512, rfktr_bontrex-150, rfktr_bwmnga, rfktr_circrex, RFKTR_fshnsta, RFKTR_plastic, RFKTR_rootrex, rfktr_sinetempore, rmadanegative402_sd15-neg, sad512, sagbobz, selfie512, shock512, slpashter_2, smile512, Style-Empire, Style-Empire-neg, style-hamunaptra, style-psycho, style-widow, suckhertonguealexV2.0, suckhertonguealexzuov4, suckhertonguealexzuov5, sxz-raiden-mgr, the_trump, ulzzang-6500, ulzzang-6500-v1.1, UnrealisticDream, wholesomegrandpas, wholesomegrannies
Textual inversion embeddings skipped(23): AnalogFilm768-BW-Classic, AnalogFilm768-BW-Modern, AnalogFilm768-BW-Tintype, AnalogFilm768-BW-Vintage, AnalogFilm768-Old-School, AnalogFilm768, Cinema768-Analog, Cinema768-BW, Cinema768-Classic, Cinema768-SilentFilm, FloralMarble-150, FloralMarble-250, FloralMarble-400, FloralMarble, nartfixer, Neg_Facelift768, nfixer, no_unrealistic768, nrealfixer, style-hamunaptra_sd2, style-widow_sd2, TungstenDispo, ZphyrInsanity_Negative-neg
Model loaded in 8.0s (load weights from disk: 0.3s, create model: 0.4s, apply weights to model: 1.7s, apply half(): 0.3s, move model to device: 0.7s, load textual inversion embeddings: 2.4s, calculate empty prompt: 2.2s).
Image Browser: ImageReward is not installed, cannot be used.
Applying attention optimization: Doggettx... done.
*Deforum ControlNet support: enabled*
CUDA SETUP: Loading binary C:\AI\stable-diffusion-webui\venv\lib\site-packages\bitsandbytes\libbitsandbytes_cudaall.dll...
preload_extensions_git_metadata for 28 extensions took 1.84s
Running on local URL:  http://0.0.0.0:7777

To create a public link, set `share=True` in `launch()`.
Startup time: 40.9s (import torch: 1.8s, import gradio: 1.4s, import ldm: 0.4s, other imports: 1.0s, setup codeformer: 0.1s, list SD models: 2.4s, load scripts: 27.2s, create ui: 1.9s, gradio launch: 4.3s, add APIs: 0.2s).
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 10.79it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 11.54it/s]
2023-06-29 22:11:31,210 - ControlNet - INFO - Loading model: control_v11p_sd15_canny [d14c016b][00:01<00:00, 12.35it/s]
2023-06-29 22:11:32,070 - ControlNet - INFO - Loaded state_dict from [C:\AI\stable-diffusion-webui\models\ControlNet\control_v11p_sd15_canny.pth]
2023-06-29 22:11:32,071 - ControlNet - INFO - Loading config: C:\AI\stable-diffusion-webui\models\ControlNet\control_v11p_sd15_canny.yaml
2023-06-29 22:11:34,544 - ControlNet - INFO - ControlNet model control_v11p_sd15_canny [d14c016b] loaded.
2023-06-29 22:11:34,616 - ControlNet - INFO - Loading preprocessor: canny
2023-06-29 22:11:34,616 - ControlNet - INFO - preprocessor resolution = 512
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  7.37it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:03<00:00,  6.40it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 10.54it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 11.22it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 10.90it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 11.11it/s]
[Tiled Diffusion] upscaling image with 4x-Fabric...████████████████████████████████████| 20/20 [00:01<00:00, 12.05it/s]
[Tiled Diffusion] ControlNet found, support is enabled.
2023-06-29 22:12:18,011 - ControlNet - INFO - Loading model: control_v11f1e_sd15_tile [a371b31b]
2023-06-29 22:12:18,815 - ControlNet - INFO - Loaded state_dict from [C:\AI\stable-diffusion-webui\models\ControlNet\control_v11f1e_sd15_tile.pth]
2023-06-29 22:12:18,815 - ControlNet - INFO - Loading config: C:\AI\stable-diffusion-webui\models\ControlNet\control_v11f1e_sd15_tile.yaml
2023-06-29 22:12:21,123 - ControlNet - INFO - ControlNet model control_v11f1e_sd15_tile [a371b31b] loaded.
2023-06-29 22:12:21,198 - ControlNet - INFO - Loading preprocessor: tile_resample
2023-06-29 22:12:21,198 - ControlNet - INFO - preprocessor resolution = 1024
MultiDiffusion Sampling:   0%|                                                                  | 0/20 [00:00<?, ?it/s]MultiDiffusion hooked into 'Euler a' sampler, Tile size: 96x96, Tile batches: 1, Batch size: 8. (ext: ContrlNet)
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:19<00:00,  1.03it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:20<00:00,  1.00s/it]
[Tiled Diffusion] upscaling image with 4x-Fabric...████████████████████████████████████| 20/20 [00:20<00:00,  1.05it/s]
[Tiled Diffusion] ControlNet found, support is enabled.
2023-06-29 22:13:20,712 - ControlNet - INFO - Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
2023-06-29 22:13:20,720 - ControlNet - INFO - Loading preprocessor: tile_resample
2023-06-29 22:13:20,721 - ControlNet - INFO - preprocessor resolution = 2048
                                                                                                                       MultiDiffusion hooked into 'Euler a' sampler, Tile size: 128x128, Tile batches: 2, Batch size: 8. (ext: ContrlNet)?it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [04:06<00:00, 12.34s/it]
MultiDiffusion Sampling:   0%|                                                                  | 0/20 [05:18<?, ?it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [02:07<00:00,  6.38s/it]
[Tiled Diffusion] upscaling image with TGHQFace8x_500k...██████████████████████████████| 20/20 [02:07<00:00,  6.28s/it]
[Tiled Diffusion] ControlNet found, support is enabled.
2023-06-29 22:20:17,114 - ControlNet - INFO - Loading model from cache: control_v11f1e_sd15_tile [a371b31b]
2023-06-29 22:20:17,128 - ControlNet - INFO - Loading preprocessor: tile_resample
2023-06-29 22:20:17,128 - ControlNet - INFO - preprocessor resolution = 2048
MultiDiffusion Sampling:   0%|                                                                  | 0/40 [00:00<?, ?it/s]MultiDiffusion hooked into 'DPM++ SDE Karras' sampler, Tile size: 128x128, Tile batches: 2, Batch size: 8. (ext: ContrlNet)
MultiDiffusion Sampling:  30%|█████████████████                                        | 12/40 [03:32<08:11, 17.56s/it]
Total progress:  37%|████████████████████████▌                                         | 13/35 [03:29<06:23, 17.43s/it]
 34%|████████████████████████████                                                      | 12/35 [03:30<06:43, 17.53s/it]

Additional information

using Nvidia drivers 536.40 w/ RTX 3090

GPU memory seems to be really loaded. not sure why. wasn't happening before.

[alenknight]

note... this is happening with a pretty standard 2048x1536 for a 2x upscale with tiling.
it's similiar with any model though. VRAM gets blown out even though i have 24GB available.

[huchenlei]

AUTOMATIC1111/stable-diffusion-webui#11063.

There is nothing we can do here in this repo. The solution seems to be downgrading?

[alenknight]

AUTOMATIC1111/stable-diffusion-webui#11063.

There is nothing we can do here in this repo. The solution seems to be downgrading?

Thanks. Odd only cause I didn’t have issues till today and I just upgraded today. But I was beyond the versions folks mention.

So that 531.x version of drivers they suggest seems to be running well for most?
I don’t mind downgrading at all.