-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Raise FileExistsError when running multiple workers #2039
Comments
I am also encountering this (luigi 2.6.1), looks like a race here when the earlier check for path existence had returned false but another task creates the path first, os.makedirs() call fails due to the leaf folder existing. |
Ah, yes, the bad things with non-atomicity. Luckily this isn't super-severe as next luigi run the tasks will be retried. Anyhow, if we can make it atomic it would be great. :) |
In Python 3.2 and above, we can use the exist_ok=True flag to os.makedirs(). In Python 2.7 I guess we have to catch the error, check errno, and reraise if errno != EEXIST. |
FileExistsError has the errno attribute.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions. |
When running multiple workers, mkdir in FileSystemTarget(LocalTarget).temporary_path() raises FileExistsError sometimes.
My output files of parallel tasks have same parent directory.
It seems that "raise_if_exists=False" of LocalFileSystem.mkdir does not work properly when running multiple workers.
test_luigi_bug.py:
I execuded this command:
PYTHONPATH='.' luigi --local-scheduler --module test_luigi_bug TestTask2 --workers 4 > luigi_bug.txt 2>&1
and I got this output:
luigi_bug.txt
Version
The text was updated successfully, but these errors were encountered: