Dense layer without activation?

[owen8877]

In

ChromaGAN/SOURCE/ChromaGAN.py

Line 209 in 30e975b

global_features2 = keras.layers.Dense(1024)(global_features2)

there is no activation parameter designated in the Dense layer. In keras 2.3.1, the default activation is linear activation (i.e. no activation):

tf.keras.layers.Dense(
    units,
    activation=None,
    ...

[motionlife]

In

ChromaGAN/SOURCE/ChromaGAN.py

Line 209 in 30e975b

global_features2 = keras.layers.Dense(1024)(global_features2)

there is no activation parameter designated in the Dense layer. In keras 2.3.1, the default activation is linear activation (i.e. no activation):

tf.keras.layers.Dense(
    units,
    activation=None,
    ...

Just want to send the class semantics for colorization head(a.k.a the decoder), so only logits is enough, probably more stable than a normalized(softmax) result.

[owen8877]

But it makes no sense to have multiple fully-connected hidden layers without activation - that is equivalent to (or potentially less than) just one fully-connected layer.

[motionlife]

@owen8877 yes, your are right, just take another look at the code, even the classification head's dense layers didn't have activation, consider that orignal vgg has relu for both the 4096-dense layers.

[owen8877]

@motionlife That's true. I hope they can fix the mistake and probably yield better performance.

[motionlife]

@owen8877 Or just use one dense layer to make model smaller

[owen8877]

@motionlife Well, we might as well stick to the vanilla VGG16 design since there is a classification loss against the pre-trained VGG16 model.

ChromaGAN/SOURCE/ChromaGAN.py

Line 149 in 30e975b

outputs=[ predAB, classVector, discPredAB])

I doubt there might be a performance recession if we cut the FC layers thin.

[motionlife]

@motionlife yes understand, I mean if there is no performance boost when add back activation. As you said, why not just use one dense.