Skip to content

Commit

Permalink
Fix a few issues in BatchLogging.groovy
Browse files Browse the repository at this point in the history
1. Adds the projectId to BatchLogging's LoggingOptions so the logs
will be fetched from the correct project. Previously I got a lot of
"that resource might not exist" responses from the log read requests.
I'm not sure if there's something special about my environment that
causes me to need the projectId in the options, but I think it
couldn't hurt to have the projectId in general.

2. Removes the old paseOutput function, which relied on the STDERR and
STDOUT prefixes in the payload of the log entries. Batch no longer adds
those prefixes so that parsing doesn't work. Now we can distinguish
stderr from stdout by looking at the logEntry's severity.

3. Catches any exception thrown by the log reading code. It could be a
permissions issuee like I was seeing, or a thread pool issuee as in
nextflow-io#3166. Either way,
something going wrong while trying to read task logs should probably
not stop the whole workflow.

With these changes, the nf-core/methylseq workflow can be run with
the google-batch executor.
  • Loading branch information
aaronegolden committed Nov 29, 2022
1 parent ae95d90 commit a8502c8
Showing 1 changed file with 13 additions and 31 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,15 @@ import com.google.cloud.logging.Logging
import com.google.cloud.logging.LoggingOptions
import groovy.transform.Memoized
import groovy.transform.PackageScope
import groovy.util.logging.Slf4j
import nextflow.cloud.google.batch.client.BatchConfig
/**
* Batch logging client
*
* @author Paolo Di Tommaso <[email protected]>
*/
@Slf4j
class BatchLogging {

private static final String STDOUT = 'STDOUT: '
private static final String STDERR = 'STDERR: '

private String mode = STDOUT
private LoggingOptions opts
private String projectId

Expand All @@ -45,8 +42,8 @@ class BatchLogging {

BatchLogging(BatchConfig config) {
final creds = config.googleOpts.credentials
this.opts = LoggingOptions .newBuilder() .setCredentials(creds) .build()
this.projectId = config.googleOpts.projectId
this.opts = LoggingOptions .newBuilder() .setCredentials(creds) .setProjectId(this.projectId) .build()
}

String stdout(String jobId) {
Expand All @@ -57,45 +54,30 @@ class BatchLogging {
return fetchLogs(jobId)[1]
}

@PackageScope String currentMode() { mode }

@Memoized(maxCacheSize = 1000)
@PackageScope List<String> fetchLogs(String uid) {
final stdout = new StringBuilder()
final stderr = new StringBuilder()
try(Logging logging = opts.getService()) {
// use logging here
final filter = "resource.type=generic_task AND logName=projects/${projectId}/logs/batch_task_logs AND labels.job_uid=$uid"
final entries = logging.listLogEntries(
Logging.EntryListOption.filter(filter),
Logging.EntryListOption.pageSize(1000) )

final stdout = new StringBuilder()
final stderr = new StringBuilder()
final page = entries.getValues()
for (LogEntry logEntry : page.iterator()) {
final output = logEntry.payload.data.toString()
parseOutput(output, stdout, stderr)
if (logEntry.severity == "ERROR") {
stderr.append(output)
} else {
stdout.append(output)
}
}

return [ stdout.toString(), stderr.toString() ]
}
}

protected void parseOutput(String output, StringBuilder stdout, StringBuilder stderr) {
// check stderr
def p = output.indexOf(STDERR)
if( p>=0 ) {
mode = STDERR
output = output.substring(p+STDERR.size())
}
else if( (p = output.indexOf(STDOUT))>=0 ) {
mode = STDOUT
output = output.substring(p+STDOUT.size())
} catch (Exception e) {
log.debug "[GOOGLE BATCH] Cannot read logs for job: `$uid` | ${e.message}"
}
// now append the result
if( mode==STDOUT )
stdout.append(output)
else
stderr.append(output)
return [ stdout.toString(), stderr.toString() ]
}

}

0 comments on commit a8502c8

Please sign in to comment.