Ability to fuse non-square (pruned) attention weights for BERT-like models

[mfuntowicz]

Following @tianleiwu implementation for non squared (i.e. pruned) attention layer this PR introduces the necessary machinery to fuse Attention layer when optimizing BERT model through onnxruntime_tools optimizer.

What does this PR does:

1. Do no assume squared matrices of shape (self.hidden_size, self.hidden_size) for QKV weights
2. Split up weights shape into a more generic in_size & out_size mapping -respectively- the incoming number of features and the projected number of features for the matmul operator.
3. Infer the hidden size representation for each individual head in order to correctly infer the dynamic number of heads in each Attention layer.
4. Correctly specify the num_heads meta attribute according point 3.

[tianleiwu]

Looks good to me.

This change supports attention head pruning (hidden size per head remains the same).

[snnn]

/azp run Windows CPU CI Pipeline

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tianleiwu]

@mfuntowicz , I triggered a wrong "Windows CPU CI Pipeline" by mistake. It blocks the merging of this pull request to master. Could you add a dummy commit (like update some comments) and see whether there are two "Windows CPU CI Pipeline" in Required Statuses? If there are still two "Windows CPU CI Pipeline" required after the commit, you will need create a new pull request instead. Sorry for the inconvenience.

[tianleiwu]

/azp run Linux CPU CI Pipeline,Linux CPU x64 NoContribops CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,MacOS CI Pipeline,MacOS NoContribops CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline

[tianleiwu]

/azp run orttraining-linux-ci-pipeline,orttraining-mac-ci-pipeline,orttraining-linux-gpu-ci-pipeline,centos7_cpu,Linux CPU Minimal Build E2E CI Pipeline,Linux Nuphar CI Pipeline,MacOS NoContribops CI Pipeline,Linux OpenVINO CI Pipeline,orttraining-distributed,orttraining-amd-gpu-ci-pipeline

[azure-pipelines]

Azure Pipelines successfully started running 9 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 10 pipeline(s).

[mfuntowicz]

@tianleiwu Is there anything I can do for all these failing pipelines? Many of them seems to be unrelated to the changes of this PR, but not sure.

Let me know 👌🏻

[tianleiwu]

@mfuntowicz, you will need integrate master to your branch so that it does not show "14 commits behind microsoft:master.". The branch missed a commit of fix for CI Pipeline: ed1883a

[mfuntowicz]

@tianleiwu I just rebased the PR on master, let me know 👍🏻