Fix initilizations

[sayakpaul]

There were a couple of inconsistencies that needed to be taken care of. The PR introduces changes to fix those.

This is how the progression of feature map sizes should look like for the test model in PyTorch:

***********RegNetYLayer***********
Hidden states: torch.Size([1, 128, 56, 56])
Residual: torch.Size([1, 128, 56, 56])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 128, 56, 56])
Residual: torch.Size([1, 128, 56, 56])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 192, 28, 28])
Residual: torch.Size([1, 192, 28, 28])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 192, 28, 28])
Residual: torch.Size([1, 192, 28, 28])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 192, 28, 28])
Residual: torch.Size([1, 192, 28, 28])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 192, 28, 28])
Residual: torch.Size([1, 192, 28, 28])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 192, 28, 28])
Residual: torch.Size([1, 192, 28, 28])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 192, 28, 28])
Residual: torch.Size([1, 192, 28, 28])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 512, 14, 14])
Residual: torch.Size([1, 512, 14, 14])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 1088, 7, 7])
Residual: torch.Size([1, 1088, 7, 7])
***********RegNetYLayer***********
Hidden states: torch.Size([1, 1088, 7, 7])
Residual: torch.Size([1, 1088, 7, 7])

Print statements were placed in the Y block's forward() method.

Currently, we are getting when running the TF integration test (playground):

Traceback (most recent call last):
  File "playground_tf_regnet.py", line 14, in <module>
    model = TFRegNetForImageClassification.from_pretrained("facebook/regnet-y-040", from_pt=True)
  File "/Users/sayakpaul/Downloads/Misc/transformers/src/transformers/modeling_tf_utils.py", line 1878, in from_pretrained
    return load_pytorch_checkpoint_in_tf2_model(model, resolved_archive_file, allow_missing_keys=True)
  File "/Users/sayakpaul/Downloads/Misc/transformers/src/transformers/modeling_tf_pytorch_utils.py", line 124, in load_pytorch_checkpoint_in_tf2_model
    return load_pytorch_weights_in_tf2_model(
  File "/Users/sayakpaul/Downloads/Misc/transformers/src/transformers/modeling_tf_pytorch_utils.py", line 155, in load_pytorch_weights_in_tf2_model
    tf_model(tf_inputs, training=False)  # Make sure model is built
  File "/Users/sayakpaul/.local/bin/.virtualenvs/hf/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/Users/sayakpaul/Downloads/Misc/transformers/src/transformers/modeling_tf_utils.py", line 383, in run_call_with_unpacked_inputs
    return func(self, **unpacked_inputs)
  File "/Users/sayakpaul/Downloads/Misc/transformers/src/transformers/models/regnet/modeling_tf_regnet.py", line 566, in call
    outputs = self.regnet(pixel_values, output_hidden_states=output_hidden_states, return_dict=return_dict)
  File "/Users/sayakpaul/Downloads/Misc/transformers/src/transformers/models/regnet/modeling_tf_regnet.py", line 373, in call
    encoder_outputs = self.encoder(
  File "/Users/sayakpaul/Downloads/Misc/transformers/src/transformers/models/regnet/modeling_tf_regnet.py", line 336, in call
    hidden_state = stage_module(hidden_state)
  File "/Users/sayakpaul/Downloads/Misc/transformers/src/transformers/models/regnet/modeling_tf_regnet.py", line 304, in call
    hidden_state = layer_module(hidden_state)
  File "/Users/sayakpaul/Downloads/Misc/transformers/src/transformers/models/regnet/modeling_tf_regnet.py", line 280, in call
    hidden_state += residual
tensorflow.python.framework.errors_impl.InvalidArgumentError: Exception encountered when calling layer "layer.0" (type TFRegNetYLayer).

Incompatible shapes: [3,55,55,128] vs. [3,111,111,32] [Op:AddV2]

Call arguments received:
  • hidden_state=tf.Tensor(shape=(3, 111, 111, 32), dtype=float32)

We need to fix this part.

[sayakpaul]

@ariG23498 the above problem is solved. During cross-loading, only the batch norm layer params (moving mean and variance) are being mismatched now. If I am able to get around with it, I will push a fix.

[sayakpaul]

Since the PyTorch model uses the activations in isolation, we need to skip the layer number.

[ariG23498]

This is a great catch!

[ariG23498]

Some pointers for me:

• Understanding when we need the keras.Sequential and when a list of layers would be fine.
• The forward() function was missed a lot of places.
• Building the main model for tensorflow.

The PR looks good to me.

[ariG23498]

@sayakpaul you can go ahead and merge it if you like!

[sayakpaul]

@ariG23498 I tried incorporating this fix: https://github.com/huggingface/transformers/pull/17571/files.

But now the warning gets changed:

['regnet.encoder.stages.1.layers.5.layer.0.normalization.num_batches_tracked', 'regnet.encoder.stages.2.layers.10.layer.3.normalization.num_batches_tracked', 'regnet.encoder.stages.1.layers.0.layer.3.normalization.num_batches_tracked', 'regnet.encoder.stages.3.layers.0.layer.3.normalization.num_batches_tracked', 'regnet.encoder.stages.2.layers.0.shortcut.normalization.moving_mean', 'regnet.encoder.stages.1.layers.0.shortcut.convolution.weight', 'regnet.encoder.stages.0.layers.0.shortcut.normalization.weight', 
...

Worth checking this with the Hugging Face folks on the main PR.