chore(ci): Start Celery worker as a background process #29371
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
github_actions
john-bodley
commented
Jun 26, 2024
| @@ -117,6 +117,20 @@ testdata() { | |||
| say "::endgroup::" | |||
| } | |||
|
|
|||
| celery-worker() { | |||
john-bodley
force-pushed
the
john-bodley--celery-worker-detached
branch
from
2d6805f to
55feb68
Compare
john-bodley
commented
Jun 26, 2024
| celery \ | ||
| --app=superset.tasks.celery_app:app \ | ||
| worker \ | ||
| --concurrency=2 \ |
There was a problem hiding this comment.
This seems clearer than -c 2.
| worker \ | ||
| --concurrency=2 \ | ||
| --detach \ | ||
| --optimization=fair |
There was a problem hiding this comment.
This seems clearer than -Ofair.
john-bodley
requested review from
villebro,
geido,
eschutho,
rusackas,
betodealmeida,
nytai,
mistercrunch,
craig-rueda,
kgabryje and
dpgaspar
as code owners
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #29371 +/- ##
==========================================
+ Coverage 60.48% 70.40% +9.91%
==========================================
Files 1931 1956 +25
Lines 76236 78405 +2169
Branches 8568 8958 +390
==========================================
+ Hits 46114 55201 +9087
+ Misses 28017 21007 -7010
- Partials 2105 2197 +92
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
dpgaspar
approved these changes
Jun 26, 2024
eschutho
pushed a commit
that referenced
this pull request
Jul 24, 2024
mistercrunch
added
🏷️ bot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
I've been fighting with inconsistencies between running CI locally and remote. Whilst working on a specific PR the Celery tests would always succeed locally whereas they would fail in a remote setting. The errors were always suppressed, but my hunch was that maybe the Celery worker(s) weren't responsive as queries remained in the pending state.
If I mangled the
python_tests.shscript to only run thetests/integration_tests/celery_tests.pytests everything seemed to be fine which indicated that maybe there was a temporal facet to the flakiness of the tests. Reading through the GitHub actions/workflows it seems like we manually background the task (via the&) yet one can actually run Celery in a detached state, which seems more robust.BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TESTING INSTRUCTIONS
CI. Also verified that the tests on my other PR successfully ran.
ADDITIONAL INFORMATION