Add author info replication script in gatys_ecker_betghe_2016

[jbueltemeier]

The replication of gatys_ecker_betghe_2016 shows clear differences in the results. For this reason, I have contacted the author. He provided helpful tips for improving the results. This information on the hyperparameters is integrated with this PR. The most important changes are:

1. the use of the activations after the ReLU function.
2. the use of layer_weights depending on the number of channels.
3. random initialisation of the image and more optimisation steps.

Due to the random initalisation, an exact replication is no longer possible, but the results show a clear improvement and a more similar stylisation to the original images.

[pmeier]

Thanks a lot for the effort Julian!

He provided helpful tips for improving the results.

Just to be sure I understand correctly: with improvement you mean better alignment of our replicated results with what is reported in the paper, right? Because that is what we are striving for. General quality improvement of the images is not a goal here.

In general, change looks good. In the future we should probably look at removing impl_params: bool in favor of config: GatysConfig like

class GatysConfig(enum.Enum):
    PAPER = enum.auto()
    REFERENCE_IMPLEMENTATION = enum.auto()
    AUTHOR = enum.auto()

This would make it much easier to deal with such a situation. This would also cover the instance_norm: bool thing that we have for Ulyanov.

[pmeier]

So here is still uncertainty and the author also couldn't give us the data he used for producing the results in the paper? Could you detail what doesn't work for 1e9?

[pmeier]

As discussed offline, the author provided these two discrete values. We'll recheck 1e9, but otherwise 1e6 is the "right" one.

[jbueltemeier]

1. I have looked at the commentary again and the ratio here refers to the weight ratio given in the paper. For this, there must be a larger value and a smaller value, and according to the author, the larger value (weight for style_loss) was the discrete value 1e6 or 1e9.
2. The optimisation is unstable with a weighting of 1e9. The following error message appears regularly:

File "/home/julian/github/pystiche_papers/.venv/lib/python3.7/site-packages/torch/autograd/__init__.py", line 175, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
RuntimeError: Function 'ConvolutionBackward0' returned nan values in its 1th output.`

[pmeier]

Thanks!