contrib/ftp: Clean up temporary files

[jethron]

Description

On Python3, the target now stores the temporary files in a tempfile.TemporaryDirectory; once the target is destroyed/garbage collected the TemporaryDirectory should be too, and remove itself and all its contents, so the temporary files don't hang around any more using disk space. A better solution would be to rewrite the module to actually try and stream the data, but this seemed nontrivial with ftplib (though possibly easier with pysftp's Connection.getfo()).

Motivation and Context

I started using the contrib/ftp targets only to discover that it doesn't actually 'stream' data as the documentation implies, but it downloads entire files to a file in /tmp which doesn't get cleaned up automatically, even after tasks complete. That worker's disk filled up and took it down temporarily.

Have you tested this? If so, how?

I'm running a similar patch and it works for me, but that one didn't keep the old behaviour for Python2 like this one does. Not sure if there's a good solution that easily works for both, but since Py2 is EOL in a few months and this behaviour is already ~5 years old, I didn't bother too much.

[dlstadther]

Thanks for this contribution! Could you verify that this code section is covered by unittests?

I added the SFTP support many years ago. Candidly, I didn't do a very good job. So feel free to improve upon it more! :)

[jethron]

Thanks @dlstadther I've added checks to a unittest and found a bug in the process. Yay tests. :)

Ran with pipenv run python -m _test_ftp & and then pipenv run python -m unittest _test_ftp.py after installing pyftpdlib.

For future people testing with virtualenvs, if you don't want to run the FTP server as root (required to bind to port 21) you can update all the port references to something higher like 2100 and it works as a normal user.

[dlstadther]

Thanks!