(Optimization) Option to remove negative conditioning at low sigma values

[devNegative-asm]

Describe what this pull request is trying to achieve.

Give users the option to change a new setting "Negative Guidance minimum sigma" where higher values give better performance by skipping the negative prompt on some sampling steps

Additional notes and description of your changes

original idea: https://twitter.com/Birchlabs/status/1640033271512702977
At low sigma values towards the latter steps of a diffusion process, there is not much practical difference between different cfg values. Setting cfg=1 for these steps allows the calls to the unet to be cut in half with minimal quality loss. This PR allows the user to choose the sigma values below which cfg will be skipped. For these steps, the uncond vector is temporarily unused to cut the batch size.
Empirical testing showed that alternating doing this every other step and increasing the sigma threshold gave better results at the same speed than without alternating, so that is fixed in the code, so every other step below the threshold will skip uncond.
The option will default to 0, meaning no change to default behavior, but users may increase it for better performance.

Environment this was tested in

• OS: Ubuntu 22.04
• Browser: Firefox
• Graphics card: NVIDIA RTX 4090 24GB

UI changes

xyz plot

Settings/Sampler parameters

output comparison

benefit
Test run on my system,
minimum sigma = 0, batch 10 512x512 150 step euler a: effective 48 it/s
minimum sigma = 1.5 batch 10 512x512 150 step euler a: effective 55 it/s
minimum sigma = 3, batch 10 512x512 150 step euler a: effective 60 it/s

[vladmandic]

i've noticed that this pr only patches k_diffusion samplers - is it applicable to compvis samplers (ddim, plms and most interestingly new unipc) ?

[devNegative-asm]

i've noticed that this pr only patches k_diffusion samplers - is it applicable to compvis samplers (ddim, plms and most interestingly new unipc) ?

You'll have to ask compvis for that. They implement CFG themselves
https://github.com/CompVis/latent-diffusion/tree/main/ldm/models/diffusion

adding it to unipc looks possible, but it's so different I don't feel confident that I could add it myself.

[vladmandic]

i've noticed that this pr only patches k_diffusion samplers - is it applicable to compvis samplers (ddim, plms and most interestingly new unipc) ?

You'll have to ask compvis for that. They implement CFG themselves https://github.com/CompVis/latent-diffusion/tree/main/ldm/models/diffusion

adding it to unipc looks possible, but it's so different I don't feel confident that I could add it myself.

@space-nuko @wl-zhao any thoughts?

[asagi4]

I've been running this for a while and it works pretty nicely, but I've noticed that any non-zero threshold can break extensions like regional prompting. I guess they don't expect the negative conditioning to just disappear on some steps, but I don't understand the logic well enough to fix it... All the tensor operations are essentially just magic to me because I have no mental model of the code flow, what the types are and how the operations relate to each other because it's all way too dynamically typed.

[AUTOMATIC1111]

this breaks generation for any prompt with AND in it