Fix false success on check

[forsyth2]

Fix false success on zstash check. Resolves #167.

[forsyth2]

@golaz This change (adding daemon=True to the multiprocessing.Process call) at least seems to make zstash fail immediately. I tested it out and it exited after two errors Exception: Error=Transferring file from HPSS: 000001.tar and Exception: Error=Transferring file from HPSS: 000000.tar, as opposed to running through all tars. However, it still prints INFO: No failures detected when checking the files., so it doesn't actually fix the bug.

Line 59 (logger.info("No failures detected when {} the files.".format(verb))) is run > failures was an empty list > extract_database returned an empty list > multiprocess_extract returned an empty list > there must be some problem in this code block:

    failures: List[FilesRow] = []
    while any(p.is_alive() for p in processes):
        while not failure_queue.empty():
            failures.append(failure_queue.get())

[forsyth2]

Note that this bug only happens when there are multiple workers. In the one-worker case, ERROR: Error=Transferring file from HPSS: 000000.tar is printed without being followed by INFO: No failures detected when checking the files.. (Interestingly, the Exception: Error=Transferring file from HPSS: 000001.tar was not printed in the one-worker case.)

[golaz]

@forsyth2 : thanks for investigating. Stopping immediately would at least be a step in the right direction.

The serial and parallel options use significantly different code, so it makes sense that they could behave differently. Based on what you have found, there might be an implementation issue with the failure queue. Unfortunately, I don't know how that code works.

Do you want to incorporate your change first as a partial solution?

[forsyth2]

Stopping immediately would at least be a step in the right direction.

@golaz Alright, should we add something like Run "grep -i Exception" on the check log to double check to INFO: No failures detected when checking the files. until we get the issue fixed fully?

I tried debugging further, but the tars that were causing the Exception suddenly appear to not cause issues. In any case, code blocks to investigate would include the following (all in https://github.com/E3SM-Project/zstash/blob/master/zstash/extract.py):

In multiprocess_extract:

    failures: List[FilesRow] = []
    while any(p.is_alive() for p in processes):
        while not failure_queue.empty():
            failures.append(failure_queue.get())

In extractFiles:

        except Exception:
            # Catch all exceptions here.
            traceback.print_exc()
            logger.error("Retrieving {}".format(files_row.name))
            failures.append(files_row)

    if multiprocess_worker:
        # If there are things left to print, print them.
        multiprocess_worker.print_all_contents()

        # Add the failures to the queue.
        # When running with multiprocessing, the function multiprocess_extract()
        # that calls this extractFiles() function will return the failures as a list.
        for f in failures:
            multiprocess_worker.failure_queue.put(f)