Add support for mixed precision in contraction scale and bilinear #973
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Extract common functionality to separate files * Reference contraction: Remove incorrect consts from type_converts * Reference contraction: Add missing type_convert for dst value * Reference contraction: Fix incorrect order of B matrix dimensions * Add support for mixed precision in contraction scale and bilinear * Move using statements from instances to a common file * Move using statements from examples to a common file * Fix the order of B matrix dimensions across examples and profiler * Fix the computation of error threshold * Make ComputeDataType an optional argument * Include possible DataType -> ComputeDataType casting error in the threshold * Remove commented code
aosewski
approved these changes
Oct 18, 2023
saadrahim
approved these changes
Oct 18, 2023
|
Please do not merge this PR. This change must be merged together with ROCm/hipTensor#141 |
LisaDelaney
approved these changes
Oct 18, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR restores the support for mixed precision in contraction scale and bilinear after it was reverted because of a conflict between CK and hipTensor.
This PR introduces support for mixed precision datatypes in the contraction operation.
It exposes a new template argument -
ComputeDataTypethat is used to cast the data before computations.The combinations supported after the change:
This change also fixes a lot of bugs/issues I found in the reference contraction, profiler and tests:
type_convertwithconstdatatypes in reference contraction;check_errfor BF16 and F16 datatypes;Normally I would prepare a separate review for every issue fix, but since GitHub does not allow for dependent PRs, it would take ages to finally merge this change. That's why I decided to put it all in this single huuuge change (it's mostly huge since I'm adding a lot of new instances, fixes themselves are mostly 1-5 liners).