Fix ComfyUI broken pipe errors seen in v0.9.13 of cog and upwards #172
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When running a ComfyUI workflow on cog v0.9.13 and upwards, the workflow would seem to randomly just stop and there would be no output. On adding error logging a broken pipe error was reported.
The error would always stop at the point where it would attempt to print or log something (like tqdm or otherwise).
ComfyUI is running in a subprocess (and the server it runs has some async methods), and the model gets updates from webhooks, which it prints. But at the same time Comfy is printing it's own outputs (like inference steps) which are also useful.
It seems to be related to this change in cog:
replicate/cog#1802
The fix here is to capture all stdout/stderr from the subprocess and log it separately.