Stable Diffusion image-to-image and inpaint using onnx.

[zledas]

Hi,

I added Stable Diffusion image-to-image and inpaint using onnx. I used non-onnx versions as "templates" and translated them according to existing text-to-image onnx.

The only problem is that this is my first time working with Python, so I was not able to use vae_encoder from onnx, so these scripts use vae that comes from "standard" Stable Diffusion for encoding. That is why if one wants to test or use these scripts, one needs to take/copy vae folder from "standard" and add

  "vae": [
    "diffusers",
    "AutoencoderKL"
  ]

to "onnx" version of model_index.json

As this is around 2.5x faster on my AMD GPU, so I think it might be useful for others as well.

Related issue: #510

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Think image-to-image is a very nice addition for ONNX. In-paint is a bit hacky at the moment. cc @anton-l

[wonderly2]

Trying to get this to work. Not sure I understood the instruction: 'one needs to take/copy vae folder from "standard"'. And I'm not sure if I'm using it properly.

`from diffusers import StableDiffusionImg2ImgOnnxPipeline
from PIL import Image

print("starting...")
pipe = StableDiffusionImg2ImgOnnxPipeline.from_pretrained("./stable_diffusion_onnx", provider="DmlExecutionProvider")

init_image = Image.open(r"start2.jpg").convert("RGB")
init_image = init_image.resize((768,512))

guidance = 7
prompt = "A fantasy landscape, trending on artstation"

image = pipe(prompt=prompt, init_image=init_image, strength=0.1, guidance_scale=guidance, num_inference_steps=25).images[0] #
image.save("stablediffusion_imgs/stablediffusion_"+str(seed)+"_"+str(guidance)+".png")
`
My error:

Traceback (most recent call last): File "C:\Users\dan\i2i.py", line 5, in <module> pipe = StableDiffusionImg2ImgOnnxPipeline.from_pretrained("./stable_diffusion_onnx", provider="DmlExecutionProvider") File "C:\Users\dan\virtualenv\lib\site-packages\diffusers\pipeline_utils.py", line 409, in from_pretrained loaded_sub_model = load_method(cached_folder, **loading_kwargs) File "C:\Users\dan\virtualenv\lib\site-packages\diffusers\modeling_utils.py", line 301, in from_pretrained model, unused_kwargs = cls.from_config( File "C:\Users\dan\virtualenv\lib\site-packages\diffusers\configuration_utils.py", line 156, in from_config config_dict = cls.get_config_dict(pretrained_model_name_or_path=pretrained_model_name_or_path, **kwargs) File "C:\Users\dan\virtualenv\lib\site-packages\diffusers\configuration_utils.py", line 212, in get_config_dict raise EnvironmentError( OSError: Error no file named config.json found in directory ./stable_diffusion_onnx.

[zledas]

Not sure I understood the instruction: 'one needs to take/copy vae folder from "standard"'. And I'm not sure if I'm using it properly.

@wonderly2 , this means that you need to copy vae folder from https://huggingface.co/CompVis/stable-diffusion-v1-4/tree/main and put it in the folder next to https://huggingface.co/CompVis/stable-diffusion-v1-4/tree/onnx stuff and add reference to it in model_index.json file. Hope this helps.

[GreenLandisaLie]

I confirm this works. Just needs to use the onnx vae and add some code to the tests i guess. Thank you @zledas for doing this. I actually did the exact same thing 1 day b4 you but didn't figure out how to work with the timesteps.

[wonderly2]

Excellent. I have it working now as well. Thanks @zledas !

[excelsior091224]

This pull request is failed to check. Why have you left it unattended for 20 days?

[patrickvonplaten]

Very sorry about being so late here! @anton-l could you please take a look here?

[anton-l]

Hi @zledas, very sorry for the late reply! If you don't mind, I'll add some commits to your PR to make it work again :)

[zledas]

Hi @zledas, very sorry for the late reply! If you don't mind, I'll add some commits to your PR to make it work again :)

@anton-l no problem, feel free to do what is needed! :)

[darktimes]

Any updates upon this?

[uchuusen]

After adding the appropriate entry into model_index.json I was able to get vae_encoder to work with:
init_latent_dist = self.vae_encoder(sample=init_image)[0]
But the next bit:
init_latents = init_latent_dist.sample(generator=generator)
fails due to there being no such object sample. I commented out that part and changed it to simply:
init_latents = init_latent_dist
and it seemed to work after that, although the seeds provided with generator no longer result in the same images as before. I'm at a loss as to how to fix that, but maybe someone else can figure it out.

[anton-l]

@zledas I've updated the SD checkpoint to include vae_encoder now, and brought the pipelines up to date with the latest features from their pytorch counterparts.
Let me know if everything works for you, and if all goes well we can merge it tomorrow! 🚀

[zledas]

@anton-l thanks for looking into and updating it! I tested the newest commit and I get this error:

2022-10-18 13:56:21.2822368 [E:onnxruntime:, sequential_executor.cc:369 onnxruntime::SequentialExecutor::Execute] Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_29' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(1978)\onnxruntime_pybind11_state.pyd!00007FFBDC8EBB9F: (caller: 00007FFBDCFFDEFF) Exception(3) tid(4304) 8007000E Not enough memory resources are available to complete this operation.

Traceback (most recent call last):
  File "D:\ML\Stable Diffusion\scripts\img2img.py", line 35, in <module>
    image = pipe(prompt=prompt, init_image=init_image, num_inference_steps=20, strength=0.75, guidance_scale=7.5, eta=0.0).images[0]
  File "D:\ML\Stable Diffusion\diffusers\src\diffusers\pipelines\stable_diffusion\pipeline_onnx_stable_diffusion_img2img.py", line 344, in __call__
    image = self.vae_decoder(latent_sample=latents)[0]
  File "D:\ML\Stable Diffusion\diffusers\src\diffusers\onnx_utils.py", line 46, in __call__
    return self.model.run(None, inputs)
  File "C:\Users\ZL\AppData\Local\Programs\Python\Python310\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 200, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_29' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(1978)\onnxruntime_pybind11_state.pyd!00007FFBDC8EBB9F: (caller: 00007FFBDCFFDEFF) Exception(3) tid(4304) 8007000E Not enough memory resources are available to complete this operation.

I'm using DmlExecutionProvider. The same setup works with my initial commits (except that I missed importing img2img in init file in my commit). Not sure if something can be done to solve this.

[darktimes]

I can confirm, that img2img works as a charm. I simply copied the content of the src folder to my virtualenv/.../diffusers folder, adjusted scripts to new pipelines and (for some reason) added vae_encoder to onnx's model.json.
Here is a test script that yielded results:

from diffusers import OnnxStableDiffusionImg2ImgPipeline
from datetime import date
from PIL import Image
import numpy as np
import random
import os

def get_latents_from_seed(seed: int, width: int, height:int) -> np.ndarray:
    # 1 is batch size
    latents_shape = (1, 4, height // 8, width // 8)
    # Gotta use numpy instead of torch, because torch's randn() doesn't support DML
    rng = np.random.default_rng(seed)
    image_latents = rng.standard_normal(latents_shape).astype(np.float32)
    return image_latents


pipe = OnnxStableDiffusionImg2ImgPipeline.from_pretrained("./stable_diffusion_onnx", provider="DmlExecutionProvider")

init_image = Image.open(r"Start.png").convert("RGB")
#init_image = init_image.resize((768,512))

prompt = "melee weapon legendary artifact, engraved blade, epic, shot from distance, 4k, hd, matte painting, by greg rutkowski"
images_to_render = 20
guidance = 7
steps = 30

#destination folder
path = os.path.join(os.getcwd(), "rendered - img2img")
if not os.path.exists(path):
    os.mkdir(path)

#create images
for x in range(images_to_render):
    seed = random.randint(0, 9999999999)
    latents = get_latents_from_seed(seed, 512, 512)
    image = pipe(prompt, num_inference_steps=steps, init_image = init_image, strength = 0.4, guidance_scale=guidance, latents=latents).images[0]
    image.save(os.path.join(path, str(seed) + ".png"))

[anton-l]

@zledas looks like it's related to what's discussed here: #791
Not sure if it's even caused by code changes, since simply reloading the pipeline seems to help with DmlExecutionProvider sometimes (#791 (comment)).

I'll look out for any developments regarding that bug though, and feel free to open an issue/PR and ping me if you find a potential solution!

[zledas]

@anton-l thanks for the link. Weird that it worked 3 times with the old commit and failed 2 times with the new (while changing commits in-between), but yeah, it looks like a deeper issue.

And thanks for cleaning, updating and committing this!

[SkyTNT]

Hello, I founed a error in scripts/convert_stable_diffusion_checkpoint_to_onnx.py . We should use OnnxStableDiffusionPipeline instead of StableDiffusionOnnxPipeline

[anton-l]

@SkyTNT thank you! Fixed it in #899

[manuforti1]

@anton-l thanks for looking into and updating it! I tested the newest commit and I get this error:

2022-10-18 13:56:21.2822368 [E:onnxruntime:, sequential_executor.cc:369 onnxruntime::SequentialExecutor::Execute] Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_29' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(1978)\onnxruntime_pybind11_state.pyd!00007FFBDC8EBB9F: (caller: 00007FFBDCFFDEFF) Exception(3) tid(4304) 8007000E Not enough memory resources are available to complete this operation.

Traceback (most recent call last):
  File "D:\ML\Stable Diffusion\scripts\img2img.py", line 35, in <module>
    image = pipe(prompt=prompt, init_image=init_image, num_inference_steps=20, strength=0.75, guidance_scale=7.5, eta=0.0).images[0]
  File "D:\ML\Stable Diffusion\diffusers\src\diffusers\pipelines\stable_diffusion\pipeline_onnx_stable_diffusion_img2img.py", line 344, in __call__
    image = self.vae_decoder(latent_sample=latents)[0]
  File "D:\ML\Stable Diffusion\diffusers\src\diffusers\onnx_utils.py", line 46, in __call__
    return self.model.run(None, inputs)
  File "C:\Users\ZL\AppData\Local\Programs\Python\Python310\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py", line 200, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_29' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(1978)\onnxruntime_pybind11_state.pyd!00007FFBDC8EBB9F: (caller: 00007FFBDCFFDEFF) Exception(3) tid(4304) 8007000E Not enough memory resources are available to complete this operation.

I'm using DmlExecutionProvider. The same setup works with my initial commits (except that I missed importing img2img in init file in my commit). Not sure if something can be done to solve this.

I have same issue. Did this get fixed? I'm using AMD Radeon RX 570 8GB so I don't see how I'm lacking resources