Codes for Figure 3 & 4 and "token_embed_c100.npy"

[wangpuyi]

I wanted to express my admiration for your impressive work! The insights and techniques demonstrated are truly noteworthy.

Additionally, could you kindly share the source code for Figures 3 and 4? Thank you for considering my request.

[d12306]

hi, @wangpuyi , it is simply by loading the embeddings and using sklearn PCA to do the dimenionality reduction and plot it.

[wangpuyi]

@d12306
Get it. Thanks for your reply.

[wangpuyi]

@d12306
By the way, could you please show the code for getting "token_embed_c100.npy"?
It will be much easier for us when using your project and potentially be beneficial of increasing your paper's influence.
Thank you for your attention.

[kxwhiowo]

@d12306
Thank you for your effort and contribution. I also wonder if it's possible for you to share the code for generating "token_embed_c100.npy".

[d12306]

Hi @kxwhiowo @wangpuyi , the token embeddings are directly obtained by indexing the tokens that corresponding to the class names in the token embedding table of the CLIP encoder in Stable Diffusion

[d12306]

I also tried get the embeddings by taking the CLIP embeddings of the corresponding class. The generated images are less intepretable. So we decided to directly go with the embeddings in the token embedding table.

[wangpuyi]

@d12306
Thanks for your thoughtful reply.
But how can we index out the token embedding out of all of the embeddings?
Says after padding, the token embeddings of "a real photo of a cat." have a shape of (1, 77, 768) [bs, tokens_num, features]. How to index out the token embedding corresponds to "cat"? Besides, it's very kind of you if the codes will be shared.
Thank you for your consideration.