Hooks Part 2 - TransformerOptionsHook and AdditionalModelsHook

[Kosinkadink]

The first Hooks PR made the code changes necessary to support hooks possible and fully added support for WeightHooks, enabling masked and scheduled weights bound to conditioning.

This PR follows up on that and cleans up the hooks implementation, adding support for two new types of hooks: TransformerOptionsHooks and AdditionalModelsHooks.

• TransformerOptionsHook: enable wrappers/callbacks/patches (i.e. things stored inside transformer_options at sample time) to be added to transformer_options either at registration time (apply to all conds) or during the sampling of the specific hooked conds; hook_scope is the variable that stores the enum corresponding to the desired behavior. The stored transformers_dict is a dict that will be merged with transformer_options. Any wrappers/callbacks/patches will be cast to the model's runtime dtype/device to comply with expectations for patches on the transformers_options within a ModelPatcher's model_options. Previously was called WrapperHook in anticipation of wrappers/callbacks/patches needing separate Hook types, but after giving it some thought my initial implementation was enough to do all transformer_options-related things.
• AdditionalModelsHook: stores a list of models to be loaded when sampling. While in the last PR I added an additional_models param to conds, the AdditionalModelsHook allows the inclusion of models to be determined at sample time, in case the model is only needed if some other hook or param on the ModelPatcher is present thanks to Hook's should_register function.

Cleanup of hook infrastructure:

• HookGroup now stores both a list of all stored hooks as well as a dict with the hooks separated by type. Enables quick parsing of only specific types in code without the need to parse all included hooks.
• With this change, HookGroup is now used instead of list[Hook] and dict[EnumHookType, list[Hook].
• Hook.clone function is now much more sane, allowing classes that inherit from a Hook class to be less annoying to write.
• Hook registration is now used to clean up any hooks on conds to make sure that by the time they reach _calc_cond_batch, there will never be a hook included that isn't intended to be used.
• What was previously called target that was just an EnumWeightTarget is now target_dict, containing an EnumWeightTarget as well as any other potential inputs. This will allow for more advanced filtering of WeightHooks for non-core features, but otherwise does not affect any previous behavior and is more of a refactor.

Fixes:

• When Cond Pair Set Props-style nodes received a hooks input while the input conditioning already had hooks applied, the combined hooks on the output conditioning did not make sure that the positive and negative conds continued to share the same hooks reference in the case that they originally did. On high VRAM cards with pairs of conds with the same hooks applied, this meant that they would instead be sampled individually, decreasing performance. On an RTX 4090 using SD1.5, in the case that there were 4 pairs of conds with each pair having the same hooks, this meant that instead of being sampled in four batches of 2 conds, they would be forced to sample as eight batches of 1 cond, dropping sampling performance up to 20%. This is now fixed.

Miscellaneous:

• Added some docstrings on Hook-related code. Will add more in the future.
• from __future__ import annotations takes away the need for the 'QuotesAroundType' workaround for typehinting in certain scenarios, so I've removed the quotes around types where the workaround would have been needed otherwise.
• Added ModelPatcher.get_injections function; should have been there in the first PR, but fell through the cracks.

Future Work:

• There will be a "Hooks Part 3" PR in the future that adds the remaining two hook types: ObjectPatchHook and InjectionsHook. With the total 5 Hook types, it will be possible to pretty much have per-cond behavior that can do anything that in the past required a model input that impacted all conds.
• Some debug nodes should be added to a ComfyOrg repo to test out all the individual wrappers/callbacks/hooks to serve as sanity checks as well as simple documentation for how to use them. WeightHooks are used by core nodes, but TransformerOptionsHooks and AdditionalModelHooks are not, so would be handy.

TL;DR: With the addition of TransformerOptionsHook and AdditionalModelsHook, custom code can now be executed on a per-cond basis by assigning wrappers/callbacks/patches to only run on specific conds, or on all conds but without the need to know the model used in sampling via a model input. Something like GLIGEN that has special ComfyUI core code to support per-cond behavior for it could instead be written using these new hooks.

[asagi4]

This sounds like it's fixing a problem I hit trying to implement scheduled attention masking. I'll try and test if I can make things work with this PR.

[asagi4]

Transformer patches get applied properly now, at least.
Here's my test implementation of an attention mask hook if anyone's interested: https://github.com/asagi4/comfyui-prompt-control/blob/attn_mask/prompt_control/nodes_attnmask.py

It's pretty slow and probably buggy, but it appears to work as expected. You can have two conds with different masks attached and it seems to be swapping the transformer options properly so that the masks don't get mixed.