This is an implementation of the paper A Neural Algorithm of Artistic Style by Leon A. Gatys, et al.. All the deep learning networks in this repository are done using PyTorch framework.
The concept of the paper is to combine the content of one image and the style of another image, in order to generate a new image which has the same content as the first image, whereas having the style of the latter. Here're some examples of this concept:
Content image: Louvre, Style image: The Great Wave off Kanagawa by Hokusai
Content image: Shanghai, Style image: The Starry Night by Vincent van Gogh
Following the original paper, I use a pretrained VGG19 model to extract image features. I use the output features from layers conv1_1, conv2_1, conv3_1, conv4_1, conv4_2, conv5_1.
The goal of this problem is to minimize the loss function, which includes the content loss between final image and content image and the style loss between final image and style image. To simplify: total_loss = alpha*content_loss + beta*style_loss, in which alpha and beta are 2 parameters we can choose. A big alpha tends to minimize content loss more, whereas a big beta will help minimize style loss more.
The content loss is computed by extracting final image's features and content's features at layer conv4_2. It is simply computed by getting the total sum of all pixels' differences between 2 extracted features.
The style loss is more complicated, it is computed by summing the style loss at conv1_1, conv2_1, conv3_1, conv4_1, conv5_1. To compute style loss for a single layer, extract the style image's features and final image's features at each layer, then compute Gram matrices of these features and calculate the sum of differences between these Gram matrices. Each layer has its own style loss weight, we can choose these parameters. In summary:
style_loss = w1*loss_conv1_1 + w2*loss_conv2_1 + w3*loss_conv3_1 + w4*loss_conv4_1 + w5*loss_conv5_1
In this repository, I use Adam to optimize final image's parameters. There're still a lot of different optimizers to try.