MaskSketch: Unpaired Structure-guided Masked Image Generation [CVPR 2023]

Jax implementation of MaskSketch.

[Paper] [Project Page] [Demo Colab]

Summary

MaskSketch is a structure-conditional image generation model based on MaskGIT. Our method leverages the structure-preserving properties of the self-attention maps of MaskGIT to generate realistic images that follow the structure given an input image or sketch.

Install the dependencies

Please use the following commands to create an environment and install the dependencies:

conda create --yes -n masksketch_env python=3.9
conda activate masksketch_env
bash install_dependencies.sh 

Running pretrained models

Class conditional Image Genration models:

Dataset	Resolution	Model	Link
ImageNet	256 x 256	Tokenizer	checkpoint
ImageNet	256 x 256	MaskGIT Transformer	checkpoint

You can run these models for sketch-conditional image generation in the demo Colab.

BibTeX

@inproceedings{bashkirova@masksketch,
    author    = {Bashkirova, Dina and Lezama, Jose and Sohn, Kihyuk and Saenko, Kate and Essa, Irfan },
    title     = {MaskSketch: Unpaired Structure-guided Masked Image Generation},
    howpublished = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year      = {2023}
}

Disclaimer

This is not an officially supported Google product.