Replace update_python_dependencies with Makefile

[garrettr]

A suggestion: we should replace update_python_dependencies with a Makefile based on the one from this blog post: A successful pip-tools workflow for managing Python package requirements.

The developer environment and workflow described in the blog post is so similar to our own that I think we could just drop the Makefile in with minimal changes. The only requirement is that we would have to add pip-tools as a dependency of the developer environment, but since we already require developers to install a set of Python dependencies as part of setting up their development environment, I think this is a totally sensible and straightforward additional requirement. The only other prerequisite is make, and since we use Makefiles elsewhere in the SecureDrop development environment, it is already a prerequisite.

Benefits of switching to a Makefile include:

• Simpler and easier to maintain
• Trivially supports more granular and developer-friendly workflows (e.g. just update one requirements file at a time)
• Automatically does "intelligent updating", only runs pip-compile when changes have been made to the corresponding .in file (this is a core feature of make)
• Almost all of update_python_dependencies is concerned with creating a virtualenv to install pip-tools in, but all of this code is unnecessary if we simply make pip-tools a requirement of the development environment.

[psivesely]

1. I know I posted that blog post on Slack or Github before. Did you find it through my posting or independently? Just curious.
2. It's funny, I was just looking at the blog post 5m ago and had the same idea. Especially now that we have a Makefile in our root directory, it seems appropriate as a central place to run such scripts.

I'm not really convinced all the bullet points in your comment enumerate benefits only or best achievable by switching to a Makefile, or that we actually.

Automatically does "intelligent updating", only runs pip-compile when changes have been made to the corresponding .in file (this is a core feature of make)

Not something we want.

Almost all of update_python_dependencies is concerned with creating a virtualenv to install pip-tools in, but all of this code is unnecessary if we simply make pip-tools a requirement of the development environment.

That reminds me that the version of update_python_dependencies that landed in develop is very slightly different than the one I intended. I'm not sure how this happened, but nonetheless the one that's in develop is fine. Part of the reason for the ephemeral virtualenv in that script (and this may not have been the original reason, it's just the reason I kept it around) is to ensure that the latest versions of pip and pip-tools are installed for reliability and so that we can take advantage of the latest features, especially relevant to securing the upgrade process both for developers and users (#1617). That said, it would be simple enough to check in the update script that these dependencies are at least at some minimum version, and exit with a message to update them if not. Tl;dr I agree it's better not to create this ephemeral virtualenv and to simply put pip-tools in the developer requirements, the virtualenv corresponding to which should be activated before running relevant scripts we'll add to the Makefile.

[garrettr]

I know I posted that blog post on Slack or Github before. Did you find it through my posting or independently? Just curious.

Independently, while I was doing research for #1800.

[garrettr]

We discussed this in today's stand-up and agreed that this is not important enough to prioritize for 0.4. Adding to a later milestone.

[redshiftzero]

See branch ill-love-pip-forever-and-always originally in PR #1923 for potentially salvageable commit