Use Pipelines without unet

[hlky]

What does this PR do?

As in #10439 this PR guards some checks involving unet in a pipeline's __init__ that prevent it from being used without unet i.e. for text encoder only.

I'll cover default values for vae_scale_factor like in #10393 for all pipelines in another PR.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sayakpaul @yiyixuxu

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sayakpaul]

Sleek!

[hlky]

Some of these checks are before register_modules and use unet, vae etc. directly, others are after register_modules and like if hasattr(self, "unet") and self.unet is not None and ..., it's ok to do either there shouldn't be any side effects from register_modules but we could consider uniformizing this.

[sayakpaul]

Fully agree!

[hlky]

Oh there's actually a reason some of these unet and scheduler checks are before register_modules, they are changing values that in turn are registered to the module's config. Others like unet.config.in_channels and default_sample_size could be after register_modules.

[sayakpaul]

Oh there's actually a reason some of these unet and scheduler checks are before register_modules, they are changing values that in turn are registered to the module's config. Others like unet.config.in_channels and default_sample_size could be after register_modules.

But I guess that doesn't block this PR no?

[hlky]

But I guess that doesn't block this PR no?

We can leave reorganising as a follow-up.

[sayakpaul]

You mean the uniformization? If so, yes, I think that is okay. That also doesn't block #10438 I think