Google Batch should comply with --Error is ignored flag #3166
Labels
Milestone
Comments
zyosufzai
changed the title
|
Can you provide the full log? There are two different tasks, |
|
Since the log are broken up by tags I have the log file for the tag [2e/9436f4] and also a trace back (attatched) |
|
Sorry the first one log file I added was for |
|
It would be great if you could upload the |
|
gotcha I couldn't find the location of the log file of that session but I reran the pipeline and directed the file to a known location. It has the same errors with the same tasks. |
|
That's a weird error. It looks reported by a thread pool used by the Google SDK |
|
So could it be that because preseq of SRR389222_sub1 and 2 failed and caused termination because the pool thread was not able to reuse the previously created threads to execute new requests? I'm wondering if the 'Ignore Errors' flag doesnt comply with google batch because it needs its own exception rule written in its JSON file? Looking at the documentations in the link below I wonder if there needs to be a "ignore_exit_status" : https://cloud.google.com/batch/docs/reference/rpc/google.cloud.batch.v1 |
aaronegolden
added a commit
to aaronegolden/nextflow
that referenced
this issue
Nov 29, 2022
1. Adds the projectId to BatchLogging's LoggingOptions so the logs will be fetched from the correct project. Previously I got a lot of "that resource might not exist" responses from the log read requests. I'm not sure if there's something special about my environment that causes me to need the projectId in the options, but I think it couldn't hurt to have the projectId in general. 2. Removes the old paseOutput function, which relied on the STDERR and STDOUT prefixes in the payload of the log entries. Batch no longer adds those prefixes so that parsing doesn't work. Now we can distinguish stderr from stdout by looking at the logEntry's severity. 3. Catches any exception thrown by the log reading code. It could be a permissions issuee like I was seeing, or a thread pool issuee as in nextflow-io#3166. Either way, something going wrong while trying to read task logs should probably not stop the whole workflow. With these changes, the nf-core/methylseq workflow runs can be run using the google-batch executor.
aaronegolden
added a commit
to aaronegolden/nextflow
that referenced
this issue
Nov 29, 2022
1. Adds the projectId to BatchLogging's LoggingOptions so the logs will be fetched from the correct project. Previously I got a lot of "that resource might not exist" responses from the log read requests. I'm not sure if there's something special about my environment that causes me to need the projectId in the options, but I think it couldn't hurt to have the projectId in general. 2. Removes the old paseOutput function, which relied on the STDERR and STDOUT prefixes in the payload of the log entries. Batch no longer adds those prefixes so that parsing doesn't work. Now we can distinguish stderr from stdout by looking at the logEntry's severity. 3. Catches any exception thrown by the log reading code. It could be a permissions issuee like I was seeing, or a thread pool issuee as in nextflow-io#3166. Either way, something going wrong while trying to read task logs should probably not stop the whole workflow. With these changes, the nf-core/methylseq workflow can be run with the google-batch executor.
aaronegolden
added a commit
to aaronegolden/nextflow
that referenced
this issue
Nov 29, 2022
1. Adds the projectId to BatchLogging's LoggingOptions so the logs will be fetched from the correct project. Previously I got a lot of "that resource might not exist" responses from the log read requests. I'm not sure if there's something special about my environment that causes me to need the projectId in the options, but I think it couldn't hurt to have the projectId in general. 2. Removes the old paseOutput function, which relied on the STDERR and STDOUT prefixes in the payload of the log entries. Batch no longer adds those prefixes so that parsing doesn't work. Now we can distinguish stderr from stdout by looking at the logEntry's severity. 3. Catches any exception thrown by the log reading code. It could be a permissions issuee like I was seeing, or a thread pool issuee as in nextflow-io#3166. Either way, something going wrong while trying to read task logs should probably not stop the whole workflow. With these changes, the nf-core/methylseq workflow can be run with the google-batch executor.
aaronegolden
added a commit
to aaronegolden/nextflow
that referenced
this issue
Dec 2, 2022
1. Adds the projectId to BatchLogging's LoggingOptions so the logs will be fetched from the correct project. Previously I got a lot of "that resource might not exist" responses from the log read requests. I'm not sure if there's something special about my environment that causes me to need the projectId in the options, but I think it couldn't hurt to have the projectId in general. 2. Removes the old paseOutput function, which relied on the STDERR and STDOUT prefixes in the payload of the log entries. Batch no longer adds those prefixes so that parsing doesn't work. Now we can distinguish stderr from stdout by looking at the logEntry's severity. 3. Catches any exception thrown by the log reading code. It could be a permissions issuee like I was seeing, or a thread pool issuee as in nextflow-io#3166. Either way, something going wrong while trying to read task logs should probably not stop the whole workflow. With these changes, the nf-core/methylseq workflow can be run with the google-batch executor.
aaronegolden
added a commit
to aaronegolden/nextflow
that referenced
this issue
Dec 2, 2022
1. Adds the projectId to BatchLogging's LoggingOptions so the logs will be fetched from the correct project. Previously I got a lot of "that resource might not exist" responses from the log read requests. I'm not sure if there's something special about my environment that causes me to need the projectId in the options, but I think it couldn't hurt to have the projectId in general. 2. Removes the old parseOutput function, which relied on the STDERR and STDOUT prefixes in the payload of the log entries. Batch no longer adds prefixes so that parsing doesn't work. Now we can distinguish stderr from stdout by looking at the logEntry's severity. 3. Catches any exception thrown by the log reading code. It could be a permissions issue like I was seeing, or a thread pool issuee as in nextflow-io#3166. Either way, something going wrong while trying to read task logs should probably not stop the whole workflow. With these changes, the nf-core/methylseq workflow can be run with the google-batch executor. Signed-off-by: Aaron Golden <[email protected]>
pditommaso
added a commit
that referenced
this issue
Dec 3, 2022
1. Adds the projectId to BatchLogging's LoggingOptions so the logs will be fetched from the correct project. Previously I got a lot of "that resource might not exist" responses from the log read requests. I'm not sure if there's something special about my environment that causes me to need the projectId in the options, but I think it couldn't hurt to have the projectId in general. 2. Removes the old parseOutput function, which relied on the STDERR and STDOUT prefixes in the payload of the log entries. Batch no longer adds prefixes so that parsing doesn't work. Now we can distinguish stderr from stdout by looking at the logEntry's severity. 3. Catches any exception thrown by the log reading code. It could be a permissions issue like I was seeing, or a thread pool issuee as in #3166. Either way, something going wrong while trying to read task logs should probably not stop the whole workflow. With these changes, the nf-core/methylseq workflow can be run with the google-batch executor. Signed-off-by: Aaron Golden <[email protected]> Signed-off-by: Aaron Golden <[email protected]> Co-authored-by: Paolo Di Tommaso <[email protected]>
marcodelapierre
pushed a commit
to marcodelapierre/nextflow
that referenced
this issue
Dec 5, 2022
1. Adds the projectId to BatchLogging's LoggingOptions so the logs will be fetched from the correct project. Previously I got a lot of "that resource might not exist" responses from the log read requests. I'm not sure if there's something special about my environment that causes me to need the projectId in the options, but I think it couldn't hurt to have the projectId in general. 2. Removes the old parseOutput function, which relied on the STDERR and STDOUT prefixes in the payload of the log entries. Batch no longer adds prefixes so that parsing doesn't work. Now we can distinguish stderr from stdout by looking at the logEntry's severity. 3. Catches any exception thrown by the log reading code. It could be a permissions issue like I was seeing, or a thread pool issuee as in nextflow-io#3166. Either way, something going wrong while trying to read task logs should probably not stop the whole workflow. With these changes, the nf-core/methylseq workflow can be run with the google-batch executor. Signed-off-by: Aaron Golden <[email protected]> Signed-off-by: Aaron Golden <[email protected]> Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Marco De La Pierre <[email protected]>
l-modolo
pushed a commit
to l-modolo/nextflow
that referenced
this issue
Jan 25, 2023
1. Adds the projectId to BatchLogging's LoggingOptions so the logs will be fetched from the correct project. Previously I got a lot of "that resource might not exist" responses from the log read requests. I'm not sure if there's something special about my environment that causes me to need the projectId in the options, but I think it couldn't hurt to have the projectId in general. 2. Removes the old parseOutput function, which relied on the STDERR and STDOUT prefixes in the payload of the log entries. Batch no longer adds prefixes so that parsing doesn't work. Now we can distinguish stderr from stdout by looking at the logEntry's severity. 3. Catches any exception thrown by the log reading code. It could be a permissions issue like I was seeing, or a thread pool issuee as in nextflow-io#3166. Either way, something going wrong while trying to read task logs should probably not stop the whole workflow. With these changes, the nf-core/methylseq workflow can be run with the google-batch executor. Signed-off-by: Aaron Golden <[email protected]> Signed-off-by: Aaron Golden <[email protected]> Co-authored-by: Paolo Di Tommaso <[email protected]>
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
Closing this in favour of #3772 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug report
Google Batch stops running even though nextflow has supplied a --Error is ignored flag
Expected behavior and actual behavior
Google Batch should continue running even if one of the jobs fails if it has been told to ignore the error. But what actually happens is the workflow terminates with a error message (see below)
Steps to reproduce the problem
Input the following on the command line:
with the following config file:
Program output
Environment
Additional context
(Add any other context about the problem here)
The text was updated successfully, but these errors were encountered: