Load the ckpts for LaViLa

[yyvhang]

Hi, thanks for opening the pre-trained ckpts. I want to know is it possible to load the checkpoint here for the model in LaViLa?
I check the 'state_dict', but it seems that the two do not correspond. Could you please provide a simple script to implement this loading?
Thanks!

[zhaoyue-zephyrus]

hey @yyvhang

Yea it's technically possible to plug the checkpoint in LaViLa. To do so, you also need to add this model config into their model as well into the LaViLa. Now LaViLa only support OpenAI's VisionTransformer (this has some key mismatch with the AVION's VisionTransformer due to some memory-efficient improvement) and LaViLa's SpaceTimeTransformer.