Truncate traceback for failure email #3086
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dlstadther
reviewed
Jun 18, 2021
DoriRunyon
force-pushed
the
truncate-error-email-stack-trace
branch
4 times, most recently
from
bf0d484 to
75501fc
Compare
|
Updated @dlstadther-pl. Please let me know if this needs any other changes. |
dlstadther
previously approved these changes
Jun 26, 2021
tnsetting
reviewed
Aug 18, 2021
DoriRunyon
force-pushed
the
truncate-error-email-stack-trace
branch
3 times, most recently
from
5a67a80 to
b565a58
Compare
DoriRunyon
force-pushed
the
truncate-error-email-stack-trace
branch
from
b565a58 to
ed39ba3
Compare
dlstadther
approved these changes
Sep 15, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add traceback_max_length config option which truncates the traceback included in task failure email (default 5000).
Motivation and Context
Suggesting this change because of an error I encountered: "ClientError: An Error occurred (InvalidParameterValue) when calling the SendRawEmail operation: message length is more than 10485760 bytes long". The root cause was a very long sql query which led to a mysql error. The error traceback included the super long query and due to its length, the luigi "task failure" email failed causing the worker to error during "_handle_next_task". The issue was then disguised, the scheduler continued to show the task as "running". The email that failed was ~17m bytes long. This adds a traceback_max_length config option for the failure email which truncates the traceback included in task failure email (default 5000 which is roughly where Rollbar truncated it).
Have you tested this? If so, how?
I included a unit test