Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debug false success in parallel check/extract #175

Open
forsyth2 opened this issue Dec 13, 2021 · 0 comments
Open

Debug false success in parallel check/extract #175

forsyth2 opened this issue Dec 13, 2021 · 0 comments
Assignees
Labels
Non-reproducible bug Bug that can't be reproduced consistently

Comments

@forsyth2
Copy link
Collaborator

forsyth2 commented Dec 13, 2021

Initially reported in #167 with partial fix in #168.

zstash check was reporting success even if exceptions occurred. This is related to the multiprocessing. Code blocks to investigate would include the following (all in https://github.com/E3SM-Project/zstash/blob/master/zstash/extract.py):

In multiprocess_extract:

    failures: List[FilesRow] = []
    while any(p.is_alive() for p in processes):
        while not failure_queue.empty():
            failures.append(failure_queue.get())

In extractFiles:

        except Exception:
            # Catch all exceptions here.
            traceback.print_exc()
            logger.error("Retrieving {}".format(files_row.name))
            failures.append(files_row)
    if multiprocess_worker:
        # If there are things left to print, print them.
        multiprocess_worker.print_all_contents()

        # Add the failures to the queue.
        # When running with multiprocessing, the function multiprocess_extract()
        # that calls this extractFiles() function will return the failures as a list.
        for f in failures:
            multiprocess_worker.failure_queue.put(f)
@forsyth2 forsyth2 added the semver: bug Bug fix (will increment patch version) label Dec 13, 2021
@forsyth2 forsyth2 self-assigned this Dec 13, 2021
@forsyth2 forsyth2 added Non-reproducible bug Bug that can't be reproduced consistently and removed semver: bug Bug fix (will increment patch version) labels Oct 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Non-reproducible bug Bug that can't be reproduced consistently
Projects
None yet
Development

No branches or pull requests

1 participant