Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(warehouse): increase async job timeout #2721

Merged
merged 7 commits into from
Nov 24, 2022
Merged

Conversation

achettyiitr
Copy link
Member

@achettyiitr achettyiitr commented Nov 22, 2022

Description

  • Increase warehouse async job timeout to 5 minutes. Since 10 seconds interval is definitely sufficient to run the Query.
  • Use a shorthand name for AsyncJobWhT as a instead of asyncWhJob.
  • Moved the global logger inside the AsyncJobWhT itself.
  • using WithConfig to populate the configs.

Notion Ticket

https://www.notion.so/rudderstacks/Warehouse-Async-Job-Framework-Sync-79a575f429e540d79e9094651b30d7e8

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@achettyiitr achettyiitr changed the title fix(warehouse): increase async job timeout to 180 seconds fix(warehouse): increase async job timeout Nov 22, 2022
Copy link
Contributor

@chaitanyapraveen chaitanyapraveen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -91,5 +91,5 @@ const (
MaxQueryRetries int = 3
RetryTimeInterval = 10 * time.Second
MaxAttemptsPerJob int = 3
WhAsyncJobTimeOut = 10 * time.Second
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make this configurable? Is this the default value we are changing?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added these.

@codecov
Copy link

codecov bot commented Nov 24, 2022

Codecov Report

Base: 46.38% // Head: 46.80% // Increases project coverage by +0.41% 🎉

Coverage data is based on head (f216d6e) compared to base (87a5d02).
Patch coverage: 0.00% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2721      +/-   ##
==========================================
+ Coverage   46.38%   46.80%   +0.41%     
==========================================
  Files         295      297       +2     
  Lines       48610    48696      +86     
==========================================
+ Hits        22550    22791     +241     
+ Misses      24646    24477     -169     
- Partials     1414     1428      +14     
Impacted Files Coverage Δ
warehouse/jobs/handlers.go 0.00% <0.00%> (ø)
warehouse/jobs/runner.go 0.00% <0.00%> (ø)
warehouse/jobs/utils.go 11.36% <ø> (+3.03%) ⬆️
warehouse/warehouse.go 9.46% <ø> (+1.03%) ⬆️
services/db/recovery.go 50.60% <0.00%> (-2.41%) ⬇️
warehouse/schema.go 50.28% <0.00%> (ø)
warehouse/internal/api/http.go 96.42% <0.00%> (ø)
warehouse/internal/repo/staging.go 81.53% <0.00%> (ø)
router/router.go 73.68% <0.00%> (+0.03%) ⬆️
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

func (asyncWhJob *AsyncJobWhT) AddWarehouseJobHandler(w http.ResponseWriter, r *http.Request) {
pkgLogger.Info("[WH-Jobs] Got Async Job Add Request")
pkgLogger.LogRequest(r)
func (a *AsyncJobWhT) AddWarehouseJobHandler(w http.ResponseWriter, r *http.Request) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a looks like too generic. rename to asyncJob ??

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@achettyiitr achettyiitr requested a review from lvrach November 24, 2022 05:53
Copy link
Contributor

@chaitanyapraveen chaitanyapraveen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants