Image only

[ZhangFfF1]

Hi, thanks for sharing the great work！
For me, this seems a bit problematic
I would like to know what you did with the image-only dataset to generate for training!
This part of the code doesn't seem to be found in your repository.
I hope you'll get back to me.
Once again, thank you for your contribution!

[muzairkhattak]

Hi @ZhangFfF1!

Thank you for showing interest in UniMed-CLIP!

Regarding your question, we convert existing image-only datasets into multi-modal format, by generating captions using LLMs (Label to Template Caption Generation technique), as described in Sec. 3.2 of our main paper.

In this repository, we only provide model training code and the dataset assembling code, alongside preprocessed annotations (image-text annotations) for image-only datasets. You can curate exact same UniMed dataset by following instructions described here at UniMed-DATA.md.

To reconstruct the captions (from scratch) for class labels, you can utilize LLMs using the prompt message given in Fig. 4 of the main paper.

Please let me know if this is helpful. Thank you and kind regards!

[ZhangFfF1]

@muzairkhattak Thank you for your reply!
For me，
I only have image data, if I use the text description generated by LLM, can it be used as part of the training set.
Hope you reply!
Thanks again nee for your excellent work!

[muzairkhattak]

Hi @ZhangFfF1,

Regarding your question, yes you can use your custom image-data and add it to UniMed training set for training.
For that, you need to convert your own image-text dataset into webdataset format, refer here for additional info on webdataset .

Once you process your dataset into webdataset format, you can simply use this new data by adding the its directory path to the train_data variable in the config file.

I hope this is helpful. Let us know if there is still any problem. Thank you!