Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Orchestrator worker crashes if accessing getLabelResolversAsync concurrently #6667

Merged
merged 16 commits into from
Apr 6, 2021

Conversation

taicchoumsft
Copy link
Member

@taicchoumsft taicchoumsft commented Apr 2, 2021

Description

Remove promise.all when calling getLabelResolversAsync, it occasionally crashes the Orchestrator worker thread.

Exceptions were also not forwarded correctly, when exceptions occurred during build or warmup of Orchestrator, the process never ends since we never send a message back to the main thread. It's also not possible to send the error object in raw form, the process boundary prevents this. So extract the message and stack property only and forward those.

Task Item

fixes #6666

@taicchoumsft taicchoumsft added P0 Must Fix. Release-blocker Area: Orchestrator labels Apr 2, 2021
@taicchoumsft taicchoumsft added this to the R13 milestone Apr 2, 2021
@taicchoumsft taicchoumsft self-assigned this Apr 2, 2021
@coveralls
Copy link

coveralls commented Apr 4, 2021

Coverage Status

Coverage increased (+0.03%) to 51.681% when pulling e7abaa7 on tachou/fixOrchWorkerExceptionHandling into 77820e6 on main.

@taicchoumsft taicchoumsft merged commit 80d4a1c into main Apr 6, 2021
@taicchoumsft taicchoumsft deleted the tachou/fixOrchWorkerExceptionHandling branch April 6, 2021 21:29
@cwhitten cwhitten mentioned this pull request May 20, 2021
lei9444 pushed a commit to lei9444/BotFramework-Composer-1 that referenced this pull request Jun 15, 2021
…concurrently (microsoft#6667)

Co-authored-by: Soroush <[email protected]>
Co-authored-by: Dong Lei <[email protected]>
Co-authored-by: Chris Whitten <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Orchestrator P0 Must Fix. Release-blocker
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Orchestrator: Warmup can crash when trying to access getLabelResolverAsync concurrently
5 participants