Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ValueError("Attempting to unscale FP16 gradients.") #127

Closed
ayush0x00 opened this issue Sep 24, 2023 · 3 comments
Closed

ValueError("Attempting to unscale FP16 gradients.") #127

ayush0x00 opened this issue Sep 24, 2023 · 3 comments

Comments

@ayush0x00
Copy link

I was trying to train AdaFace on a custom dataset with 10k classes. When the model started to train, I got a ValueError(Attempting to unscale FP16 gradients). It's obvious that FP16 gradients can't be scaled and unscaling/scaling is handled internally by AMP but I am not able to find the root cause of the error. I have also attached a screenshot of the same.

Screenshot 2023-09-24 at 6 19 33 PM
@ayush0x00 ayush0x00 mentioned this issue Sep 24, 2023
Open
@ayush0x00
Copy link
Author

Got resolved. Closing the issue.

@afm215
Copy link

afm215 commented Sep 24, 2023

Can you please explain what was happening to help people who may meet this issue in the future?

@ayush0x00
Copy link
Author

I accidentally modified the original code PReLU layer to use a torch.Float16 datatype which was causing the issue. The original code didn't have any issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@afm215 @ayush0x00