txt+image as input generating image

[Youngon]

Could this model be used in such situation, user input a prompt along with a image, generate a result image.
for example:
prompt[generate a image suck like the input image] + image[IMG.png] -> generated image

[MqLeet]

I also wonder whether Janus-Pro-7B has this ability!

[Youngon]

I have changed the pipeline and can do "txt+img -> img" work, but the model doesn't work well, so looking up to the upgrade from janus.

[MqLeet]

I have changed the pipeline and can do "txt+img -> img" work, but the model doesn't work well, so looking up to the upgrade from janus.

Hi @Youngon , could you please share your modified "txt+img -> img" pipeline and corresponding preprocess codes? Thank you!🤗

[Youngon]

I checked my pipeline, if masking the image embeddings, it can generate image according to the prompt; while I add unstanding part, can successfully unstand the content of input image.

so the model doesn't have the abblity to do "txt + img -> img" task.

[Youngon]

I have changed the pipeline and can do "txt+img -> img" work, but the model doesn't work well, so looking up to the upgrade from janus.

Hi @Youngon , could you please share your modified "txt+img -> img" pipeline and corresponding preprocess codes? Thank you!🤗

it does not work at all, the model has no abblity to use input image generating new image.

[MqLeet]

alright, thanks for your time! 🥰